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ABSTRACT 


Since  the  end  of  the  Cold  War,  reduced  budgets  have  limited  technology  growth 
in  the  defense  industry  making  the  use  of  Commercial  Off-The-Shelf  (COTS)  software 
the  accepted  way  to  build  systems.  Twenty  years  ago,  almost  all  DOD  software-intensive 
systems  were  built  by  awarding  large  multimillion-dollar  contracts  to  defense  contractors 
to  build  systems  from  scratch.  Consequently,  with  dwindling  budgets,  the  military  has 
recognized  that  they  can  no  longer  build  an  infrastructure  independent  of  commercial 
industry. 

The  use  of  commercial  items  does  not  reduce  or  eliminate  the  risks  associated 
with  the  traditional  development  of  software  systems.  Numerous  programs  have 
stumbled  for  the  lack  of  careful  consideration  and  identification  of  the  unique  risk  factors 
imposed  by  commercial  items.  Even  though  the  types  of  programs  are  diverse,  there  are 
common  risk  factors  that  can  be  identified  from  the  past  experiences  of  these  programs. 

This  thesis  focuses  on  the  critical  risk  factors  and  lessons  learned  associated  with 
integrating  commercial  items  into  DOD  software  programs.  It  summarize  lessons  leam 
from  programs  that  have  made  extensive  use  of  commercial  items,  provides  a  risk 
checklist/questionnaire  to  assist  PMs  and  developers  in  understanding  the  risks  associated 
with  their  developments  of  a  system  using  commercial  items,  and  suggests  mitigation 
strategies,  which  can  be  used  as  guidelines  for  the  risk  factors,  to  consider  when  adopting 
commercial  components.  Providing  the  starting  point  for  a  systematic  structure  approach 
to  the  risk  management  of  commercial  items. 
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I.  INTRODUCTION 


A.  AREA  OF  RESEARCH 

Building  of  systems  from  commercial  items  depends  on  successful  evaluation  and 
selection  of  the  commercial  software.  A  number  of  risks  associated  with  commercial 
items  have  been  identified  in  the  literature.  This  thesis  will  research  the  risk  factors 
associated  with  the  use  of  commercial  items  in  Department  of  Defense  Management 
Information  Systems  (MIS);  command  and  control  systems;  and  weapons  systems.  This 
will  provide  valuable  insight  into  reducing  the  risks  associated  with  COTS-Based 
Systems  (CBS)  development. 

B.  RESEARCH  QUESTIONS 

Principal  Research  Question:  What  are  the  unique  challenges  and  risk  factors  that 
need  to  be  managed  when  selecting  COTS  products  for  Department  of  Defense  Systems 
(DOD)? 

Secondary  Research  Questions: 

•  What  are  the  definitions  of  commercial  items  and  risk  management? 

•  How  are  the  familiar  notions  of  risk  and  risk  management  affected  by  the 
presence  of  commercial  software? 

•  Is  there  a  specific  approach/process  that  addresses  risk  identification  for 
commercial  (COTS)  based  programs? 

•  What  makes  the  integration  of  commercial  (COTS)  products  into  a  system 
different  from  traditional  integration  of  items  designed  and  produced  for 
DOD? 

C.  DISCUSSION 

Throughout  the  Department  of  Defense  (DOD),  operations  and  support  costs  are 
rising,  with  fewer  dollars  available  for  research,  test  and  evaluation  and  procurement  of 
new  systems;  thus,  increasing  the  pressure  to  achieve  more  with  less.  To  achieve  this 
goal,  DOD  is  expanding  the  use  of  commercial  items  to  leverage  the  perceived  massive 
technology  investments  of  the  private  sector  while  allowing  DOD  to  reap  the  benefits  of 
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reduced  cycle  times,  faster  insertion  of  new  technologies  and  lower  life  cycle  costs, 
increasing  system  stability,  higher  number  of  alternative  solutions  and  increasing  level  of 
system  interoperability.  These  advantages  also  bring  related  disadvantages,  including 
integration  difficulties,  performance  constraints,  and  incompatibility  among  products  for 
different  vendors. 

The  use  of  commercial  items  does  not  reduce  or  eliminate  the  risks  associated 
with  the  traditional  development  of  software  systems.  Despite  the  risks,  if  COTS 
acquisition  is  addressed  correctly  it  can  provide  significant  benefits  for  buying 
commercial  software;  unfortunately  the  DOD  policy  on  the  risk  management  of 
commercial  items  is  lacking.  In  the  new  less-restricted  DOD  directive  5000.1,  the 
program  manager  is  expected  to  tailor  risk  management  practices  to  the  needs  of  the 
program.  Tailoring  DOD  risk  management  policy  to  support  commercial  items  leaves 
the  program  manager  with  too  much  guesswork.  A  program  manager  using  commercial 
items  cannot  reasonably  benefit  from  DOD  risk  management  guidance,  procedures,  and 
tools  because  he  or  she  is  focused  on  new  development  program  risks  and  risk 
management  practices.  Missing  are  any  explicit  considerations  of  unique  commercial 
items  risk  and  risk  management.  Underestimating  the  risks  associated  with  commercial 
software  has  often  resulted  in  longer  schedule  delay,  higher  development  cost,  and  higher 
maintenance  cost. 

Because  of  the  numerous  risks  inherent  in  the  use  of  commercial  items,  there 
needs  to  be  a  flexible  and  proactive  commercial  item  based  risk  management  approach  to 
avoid  common  mistakes  in  commercial  items  utilizations.  Government  system 
integrators  need  to  be  aware  of  the  differences  between  military  and  commercial 
acquisition  and  the  potential  challenges  and  risks  that  need  to  be  managed.  The  objective 
of  this  thesis  is  to  focus  on  the  critical  risk  factors  associated  with  integrating  commercial 
items  into  DOD  software  programs.  It  will  summarize  lessons  learn  from  programs  that 
have  made  extensive  use  of  commercial  items  and  offer  suggestions  for  reducing  the  risk 
of  developing  systems  with  commercial  items. 
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D.  SCOPE  OF  THESIS 

The  scope  of  the  thesis  will  include  the  following: 

•  In-Depth  review  of  available  literature,  DOD  Regulations,  Audits  and 
Reports  as  they  relate  to  the  use  of  commercial  items. 

•  Interview  Program  Managers  (PMs)  who  utilize  commercial  products 
within  their  programs  to  reflect  their  experience  and  lessons  learned  from 
using  COTS  within  their  programs. 

•  Provide  a  questionnaire  to  serve  as  a  checklist  to  identify  and  understand 
the  risk  factors  associated  with  COTS  for  the  PMs  to  complete. 

•  Summarize  and  analyze  the  questionnaires  and  various  lessons  learned 
found  in  technical  documents  about  risk  associated  commercial  items. 

•  Derive  conclusion  and  provide  suggestions  for  reducing  the  identified  risk 
factors  of  commercial  items. 


E.  METHODOLOGY 

The  methodology  used  in  this  research  consists  of  the  following  steps: 

•  Conduct  a  literature  search  of  books,  journal  articles,  magazine  articles, 
World  Wide  Web,  and  other  library  infonnation  resources  regarding  the 
definitions  and  history  of  commercial  items  and  risk  management. 

•  Analyze  and  evaluate  lessons  learned  and  audit  reports  to  identify  risk 
factors  and  challenges  (reliability,  maintainability,  and  availability) 
associated  with  COTS. 

•  Develop  a  questionnaire  based  on  these  factors  and  challenges.  Send  this 
questionnaire  electronically  and  conduct  interviews  with  Product 
Managers  in  the  Department  of  Defense  whose  programs  do  or  have  tried 
to  utilize  commercial  items 

•  Summarize  and  analyze  results  of  the  questionnaire. 

•  Propose  mitigation  suggestions  for  reducing  these  factors. 


F.  ORGANIZATION 

This  thesis  is  organized  into  the  following  sections: 

•  Chapter  II:  Commercial  Items  Defined.  The  introduction  of  COTS 
products  into  system  acquisition  has  resulted  in  new  tenns  associated  with 
this  process.  Chapter  II  provides  the  terms  and  definitions  associated  with 
commercial  items. 
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•  Chapter  III:  Risk  and  Challenges  of  Commercial  (COTS)  items.  Chapter 
III  provides  a  summary  of  different  sources,  DOD  technical  reports, 
audits,  and  handbooks,  used  for  identifying  the  unique  challenges  and 
risks  within  a  system  when  using  commercial  items. 

•  Chapter  IV:  COTS  Risk  Questionnaire/Checklist.  Chapter  IV  provides 
the  framework  for  evaluating  and  reducing  the  risk  of  software  systems 
developed  with  commercial  items.  The  key  to  successful  development  of 
any  system  is  having  a  sound  approach  and  asking  the  right  questions.  A 
critical  set  of  questions  were  developed  and  sent  to  project  managers  to 
assist  them  with  understanding  the  risks  associated  with  the  developed  of 
systems  using  commercial  items  within  DOD. 

•  Chapter  V:  Questionnaire  Implementation.  Chapter  V  sending  the 
questionnaire  electronically  to  elicit  responses  and  provide  results  on 
current  DOD  projects  using  commercial  products. 

•  Chapter  VI:  Mitigation  Strategies  and  Suggestions.  Chapter  VI  describes 
some  mitigation  strategies  and  offers  some  techniques/suggestions,  which 
can  be  used  as  guidelines  for  the  risk  factors,  to  consider  when  adopting 
COTS  components. 

•  Chapter  VII:  Conclusion.  Chapter  VII  provides  thesis  conclusion  and 
rec  ommendations . 


G.  BENEFIT  OF  THE  STUDY 

The  practice  of  risk  management  does  not  benefit  from  “cookbook”  solutions 
[STEV97].  Numerous  programs  have  stumbled  for  the  lack  of  careful  consideration  and 
identification  of  the  unique  risks  factors  imposed  by  commercial  items.  Even  though  the 
types  of  programs  are  diverse,  there  are  common  risk  factors  that  can  be  identified  from 
the  past  experiences  of  these  programs.  DOD  PMs,  developers,  and  contractors  can 
benefit  from  these  past  experiences  since  they  provide  a  starting  point  for  a  systematic 
structure  approach  to  risk  management  of  commercial  items  and  assist  them  in 
overcoming  these  barriers  within  their  programs.  They  will  be  able  to  identify,  as  early 
as  possible,  the  common  risk  factors  and  lessons  learned  from  similar  programs  and 
determine  how  to  use  these  experiences  to  adjust  strategies  and  manage  the  risks,  thus 
enhancing  their  programs  while  meeting  OSD  goals  for  incorporating  COTS  into  their 
programs. 
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II.  COMMERCIAL  ITEMS  DEFINED 


A.  INTRODUCTION 

For  the  government,  making  greater  use  of  commercial-off-the-shelf  (COTS) 
products  is  becoming  increasingly  popular.  Increased  use  of  commercial  products  holds 
the  hope  of  getting,  at  a  reasonable  cost,  something  that  already  perfonns  the  functions 
needed  by  government  systems.  Everyone  from  industry  executives  to  Congress  is 
suggesting  that  leveraging  commercial  capabilities  will  save  time  and  money  while 
improving  the  performance  of  software-intensive  systems.  To  encourage  this  approach, 
then-Secretary  of  Defense  William  Perry  directed  in  June  1994  that  DOD  acquisitions 
should  make  maximum  use  of  perfonnance  specifications  and  commercial  standards,  thus 
increasing  the  opportunities  to  make  use  of  commercial  products.  The  purpose  of  this 
section  is  to  explain  the  different  ways  that  the  tenn  COTS  has  been  used,  and  the  role 
commercial  products  can  play  in  a  commercial  (COTS)-based  system.  The  following 
definition  of  COTS  is  provided  for  the  sole  purpose  of  understanding  the  intent  and  scope 
of  this  paper.  It  is  not  meant  to  be  ‘the’  definition. 


B.  DEFINING  “COTS” 

The  term  COTS  is  a  familiar  and  well-used  term  within  industry  and  DOD.  The 
tenn  COTS  is  generally  used  to  describe  commercial  products.  It  commonly  refers  to 
things  that  one  can  buy,  ready-made,  from  some  manufacturer’s  “store  shelf’  (through  a 
catalogue  or  from  a  price  list).  This  usage,  however,  is  imprecise  and  not  universally 
accepted.  The  govermnent,  which  needs  a  precise  definition  for  procurement,  has 
defined  the  term  commercial  item.  On  26  June  2000,  the  Office  of  the  Secretary  of 
Defense  (OSD)  defined  a  commercial  item  in  their  report,  “Commercial  Item 
Acquisition:  Considerations  and  Lessons  Learn”  [DODAOO].  This  official  definition  of 
the  tenn  commercial  item  is  given  in  the  Federal  Acquisition  Regulations  (FARs), 
appendix  A;  a  summary  along  with  an  illustrative  example,  figure  1,  are  provided  here. 
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A  commercial  item  is 


(1)  Any  item,  other  then  real  property,  that  is  of  a  type  customarily  used  for 
nongovernmental  purposes  and  has  been  sold,  leased,  or  licensed,  or  offered  for  sale, 
lease  or  license  to  the  general  public; 

(2)  Any  item  evolved  from  an  item  in  (1)  through  advances  in  technology  and 
is  not  yet  available  commercially  but  will  be  available  in  time  to  satisfy  the  requirement; 

(3)  Any  item  that  would  satisfy  (1)  or  (2)  but  for  modifications  customarily 
available  in  the  commercial  marketplace  or  minor  modifications  made  to  meet  DOD 
requirements; 

(4)  Any  combination  of  items  meeting  (1),  (2),  (3)  above  or  (5),  below,  that 
are  customarily  combined  and  sold  in  combination  to  the  general  public; 

(5)  Services  for  installation,  maintenance,  repair,  training,  etc.  if  such  services 
are  procured  for  support  of  an  item  in  (1),  (2),  (3)  or  (4)  above,  as  offered  to  the  public  or 
provided  by  the  same  work  force  as  supports  the  general  public,  or  other  services  sold 
competitively  in  the  commercial  marketplace; 

(6)  Services  offered  and  sold  completively  in  the  commercial  market-place  at 
catalog  prices; 

(7)  Any  item,  combination  of  items  or  service  referred  to  in  (1)  -  (6),  above, 
that  have  been  transferred  between  or  among  separate  divisions,  subsidiaries,  or  affiliates 
of  a  contractor; 

(8)  A  nondevelopmental  item  (NDI)  developed  exclusively  at  private  expense 
and  sold  competitively  to  multiple  state  and  local  governments. 


The  key  point  is  that  commercial  product(s)  are  developed  by  a  commercial  entity 
for  commercial  purposes  and  for  the  general  public.  In  order  to  gain  the  advantages  of 
commercial  products,  commercial  software  products  should  be  defined  as  those  that  are 
offered  to  the  public  and  are  actually  used  by  the  public  in  the  same  version  as  those  in 
military  applications. 
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C.  DEFINING  NDI 

A  closely  related  term  often  heard  in  government  circles  is 
“nondevelopmental  item”  (NDI). 

A  nondevelopmental  item  is: 

(1)  Any  previously  developed  item  used  exclusively  for  government  purposes 
by  a  federal,  state,  or  local  agency  or  government  or  by  a  foreign  government  that  has  a 
mutual  defense  agreement  with  the  U.S.; 

(2)  Any  item  described  in  (1)  above  that  requires  only  minor  modification  or 
modifications  normally  available  in  the  commercial  marketplace  to  meet  requirements; 

(3)  Any  item  being  produced  that  does  not  meet  (1)  or  (2)  above  only  because 
it  is  not  yet  in  use. 

The  key  point  here  is  that  NDI  refer  to  something  already  developed  by  someone 
else.  It  might  have  been  developed  by  a  commercial  interest,  but  typically  it  will  have 
been  developed  for  some  other  government,  department,  or  agency.  Hence,  what  is 
commonly  called  “government  off-the-shelf’  (GOTS)  is  a  fonn  of  NDI  item.  A  large- 
scale  example  of  a  NDI  would  be  a  fighter  aircraft  developed  by  some  other  nation.  A 
more  meaningful  example  in  the  current  context  would  be  radar  developed  for  one 
aircraft  that  is  available  for  use  in  another  aircraft. 

D.  COMMERCIAL  ITEMS  CLARRIFIED 

On  January  5,  2001,  Dr.  Gansler,  then  Under  Secretary  of  Defense  for 
Acquisition,  Technology  and  Logistics,  issued  a  policy  memorandum  to  clarify  and  to 
help  overcome  some  of  the  barriers  being  experienced  within  the  Department  of  Defense 
in  utilizing  commercial  items.  An  Integrated  Process  Team  (IPT)  had  been  formed  at  his 
direction  and  was  headed  by  both  the  Deputy  Under  Secretary  of  Defense  for  Acquisition 
Reform  (DUSD  (AR))  and  the  Director  of  Defense  Procurement.  The  IPT  was  chartered 
to  review  DOD  commercial  item  determinations  and  evaluate  whether  additional 
guidance,  tools,  or  training  were  necessary.  Dr.  Gansler’s  memorandum  says  that  the  IPT 
found  “inconsistent  commercial  item  determination  and  weak  market  research  among  the 
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obstacles  that  exist  to  broadening  the  use  of  commercial  items  within  the  DoD.” 
[DODAOl] 

Dr.  Gansler’s  memorandum  also  provided  clarifying  definitions  of  FAR  Part  12 
for  greater  consistency  within  DoD.  Four  of  the  most  important  of  these  are  as  follows: 


Commercial  Off-the-Shelf  (COTS):  A  product  does  not  have  to  be  commercial- 
off-the-shelf  (COTS)  to  meet  the  “commercial  item”  definition.  COTS  items  are  a  subset 
of  commercial  items.  The  commercial  item  definition  is  much  broader  than  products  that 
are  presently  available  off-the-shelf.  It  includes  items  that  have  only  been  “offered”  for 
sale,  lease,  or  license  to  the  general  public,  as  well  as  those  that  have  evolved  from  a 
commercial  item  and  are  offered  for  sale,  even  if  not  yet  available  in  the  commercial 
marketplace.  However,  evolved  items  must  be  available  in  the  commercial  marketplace 
in  time  to  satisfy  solicitation  delivery  requirements.  In  addition,  all  other  elements  of  the 
commercial  item  definition  at  FAR  2.101  must  also  be  met. 


Modified  Commercial  Items:  When  items  available  in  the  commercial  market 
cannot  meet  the  Department’s  need,  DoD  must  determine  whether  market  items  can  be  or 
have  been  modified  so  that  FAR  Part  12  can  be  used.  Two  types  of  modifications  are 
available:  (1)  modifications  of  a  type  available  in  the  commercial  marketplace;  and,  (2) 
minor  modifications  of  a  type  not  customarily  available  in  the  commercial  marketplace 
made  to  Federal  Government  requirements.  For  modifications  of  a  type  available  in  the 
commercial  marketplace,  the  size  or  extent  of  modifications  is  unimportant.  For  minor 
modifications,  the  item  must  retain  a  predominance  of  nongovernmental  functions  or 
physical  characteristics. 


“Of  a  Type”:  The  phrase  “of  a  type”  is  not  intended  to  allow  the  use  of  FAR  Part 
12  to  acquire  sole-source,  military  unique  items  that  are  not  closely  related  to  items 
already  in  the  marketplace.  Instead,  “of  a  type”  broadens  the  commercial  item  definition 
so  that  qualifying  items  do  not  have  to  be  identical  to  those  in  the  commercial 
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marketplace.  The  best  value  offer  in  a  competitive  Part  12  solicitation  can  be  an  item  that 
has  previously  satisfied  the  Government’s  need  but  has  not  been  sold,  leased,  licensed, 
nor  offered  for  sale,  lease  or  license  to  the  general  public  (a  nondevelopmental  item  as 
defined  in  10  USC  403  (13).  In  this  scenario,  the  phrase  “of  a  type”  allows  the  best  value 
offer  to  qualify  for  a  Part  12  contract  as  long  as  it  is  sufficiently  like  similar  items  that 
meet  the  government’s  requirement  and  are  sold,  leased,  licensed,  or  offered  for  sale, 
lease  or  license  to  the  general  public.  In  such  instances,  “of  a  type”  broadens  the  statutory 
commercial  item  definition  to  allow  Part  12  acquisition  of  a  government-unique  item  that 
can  compete  with  commercial  items  that  meet  the  government’s  requirement.  This 
avoids  the  undesirable  result  of  shutting  out  otherwise  price-competitive  preexisting 
suppliers  of  government-unique  items  from  Part  12  solicitations.  [DODAOl] 


Government  Off-The-Shelf  (GOTS)  is  also  a  commonly  used  tenn  for 
nondevelopmental  items  (NDI)  that  are  government-unique  items  in  use  by  federal,  state 
or  other  governmental  agency  or  by  a  foreign  government  with  which  the  United  States 
has  a  mutual  defense  cooperation  agreement.  This  area  will  also  be  excluded  and  this 
tenn  will  not  be  utilized  in  this  thesis. 


Since  COTS  has  been  defined  as  a  subset  of  “commercial  items”  and  Dr. 
Gansler’s  memorandum  specifically  addresses  the  broader  scope  of  commercial  items, 
this  researcher  will  use  the  tenn  “commercial  item(s)”  throughout  this  thesis. 
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Commercial 

Item 


developmental 

Item 


(i) 

Any  previously  developed 
item  used  by  federal,  state, 
local,  or  allied  governments 


(1)  that  require  only  minor 
modification 


Integration  of  NDI 
subsystems  and  components 


Figure  1.  FAR  Definition  Summarized  “From  [DODA96]” 
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E  COMPARING  COMMERCIAL  (COTS)  ITEMS  AND  NDI 

NDI  and  commercial  items  are  similar  in  that  they  both  already  exist,  which  is 
what  makes  them  attractive.  They  are  different  in  that  commercial  products  usually 
appear  in  some  sort  of  catalogue  or  price  list,  whereas  it  may  be  more  difficult  to  discover 
the  existence  of  NDI.  Commercial  items  used  to  be  considered  a  subset  of  NDI,  but  now 
this  position  is  reversed,  since  a  restricted  fonn  of  NDI  qualifies  as  a  commercial  item 
(see  item  (8)  under  commercial  item).  In  addition,  support  services  such  as  installation 
and  training  are  now  also  defined  as  commercial  items. 

Because  the  specific  emphasis  within  DoD  is  to  increase  the  utilization  of 
commercial  items,  and  nondevelopmental  items  (NDI)  are  placed  second  in  the  priority 
of  implementation,  this  thesis  will  only  address  commercial  item  usage.  NDI  literature 
will  be  reviewed  inasmuch  as  problems  experienced  utilizing  NDI,  will  for  the  most  part, 
apply  to  commercial  item  usage  as  well.  Also,  there  is  a  great  deal  of  indiscriminate  and 
imprecise  terminology  usage.  Thus,  when  the  term  “NDI”  is  used,  the  literature  may 
actually  be  referring  to  a  commercial  item.  Thus,  the  same  types  of  precautionary  and 
risk-mitigation  suggestion  recommended  in  this  thesis  for  commercial  items  may  be  used 
when  acquiring  nondevelopmental  items. 

F.  WHAT  ARE  COMMERCIAL  (COTS)-BASED  SYSTEMS  (CBS)? 

A  commercial  (COTS)-based  system  is  a  system  that  has  been  built  primarily  by 
acquiring  and  assembling  a  set  of  components  that  are  commercial  products  into  a 
working  system.  These  components  may  perform  generic  functions  that  are  independent 
of  the  system’s  application  domain  or  services  that  other  components  of  the  system 
depend  on. 

Incorporating  commercial  software  products  to  form  a  commercial-based  system 
involves  designing  their  interfaces  with  other  system  components,  and  the  work  of 
integrating  them.  The  integrator  responsible  for  building  the  system  does  so  by  buying 
the  components  and  assembling  them  into  a  complete  system.  This  involves  minimal 
code  development  as  most  of  the  components  are  not  developed  from  source  but  are 
purchased  off-the-shelf.  The  code  development  required  is  that  necessary  to:  tailor  the 
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commercial  software;  build  the  components  that  are  not  being  supplied  through 
commercial  sources;  and  glue  the  commercial  components  together.  The  number  of 
interfaces  a  commercial  product  has  with  other  system  components,  and  the  number  of 
components  it  interfaces  with  determines  the  extent  of  integration. 

One  type  of  system  is  the  commercial  solution,  a  system  in  which  a  single 
commercial  product  or  product  suite,  provided  by  one  vendor,  that  may  be  tailored  that 
provides  the  primary  solution  to  the  problem.  The  amount  of  tailoring  and  data 
conversion  is  often  significant.  These  systems  may  be  found  in  application  areas  with 
general  concurrence  on  application  practices,  examples  being  personnel  management  and 
financial  management  applications. 

A  commercial  intensive  system,  commercial-aggregate  system,  one  in  which  a 
number  of  commercial  components  have  been  acquired  from  different  sources  and  are 
assembled  into  a  complete  system.  Combining  the  functionality  of  the  different 
components  provides  the  system  services;  there  is  no  dominant  single  commercial 
component.  Often  the  use  of  the  particular  set  of  components  is  unprecedented  and 
requires  substantial  resources  to  select  and  integrate  a  cohesive  set  of  components.  These 
systems  may  be  found  where  the  needs  of  the  system  cannot  be  satisfied  by  a  single 
product  or  product  suite  or  when  the  system’s  operational  procedures  are  unique.  For 
example,  a  command  and  control  system  could  be  constructed  from  a  client/server  GUI 
system,  a  GIS  system,  a  set  of  databases,  and  data  analysis  tools. 

G.  RISK  MANAGEMENT 

Risk  Management,  one  of  the  most  critical  and  most  difficult  aspects  for  project 
management,  is  a  proactive  approach  for  minimizing  the  uncertainty  and  potential  loss 
associated  with  a  project.  It  follows  a  two-stage,  repeatable  and  iterative  process  of 
assessment  (i.e.,  the  identification,  estimation  and  evaluation  of  the  risks  confronting  a 
program)  and  management  (i.e.,  the  planning  for,  monitoring  of,  and  controlling  of  the 
means  to  eliminate  or  reduce  the  likelihood  or  consequences  of  the  risks  discovered).  It 
is  performed  continually  over  the  life  of  a  program,  from  initiation  to  retirement. 
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The  unpredictable  quality  of  commercial  software  creates  a  unique  set  of  risks  for 
software  systems  using  commercial  components.  The  CBS  acquisition  process,  then, 
should  include  risk  management,  which  identifies  high-risk  items  that  can  jeopardize 
system  quality  and  attempts  to  resolve  them  as  early  as  possible  to  ensure  rapid  and 
successful  delivery  of  the  system.  Early  identification  and  understanding  the  risks 
associated  with  commercial  items  is  the  first  step  to  ensuring  that  the  acquiring  activities 
can  achieve  the  benefits  of  using  commercial  products. 

This  thesis  will  focus  on  this  first  step  of  risk  management,  the  assessment, 
identifying  the  unique  problems,  risk  factors  associated  with  integrating  commercial 
items  that  have  an  important  influence  on  whether  a  program  will  succeed  or  fail.  It  will 
provide  the  understanding  of  these  unique  risks  to  the  program  managers  so  they  can 
successfully  execute  their  programs  and  manage  the  risks  associated  with  commercial 
items.  It  will  also  provide  some  suggestions  for  mitigating  these  factors.  A  wise 
manager  will  recognize  risks  that  are  specific  to  the  integration  of  multiple  products  into 
a  CBS  and  will  take  action  to  mitigate  and  control  those  risks. 

H.  SUMMARY 

This  chapter  has  explored  the  definition  of  commercial  items  and  commercial 
based  systems.  It  uses  the  broader  term  of  commercial  items  from  the  FAR  when 
referring  to  COTS.  A  CBS  was  defined  as  a  system  that  has  been  built  primarily  by 
acquiring  and  assembling  either  a  single  component  or  a  set  of  components  that  are 
commercial  products  and  integrating  them  into  a  working  system.  In  order  for  systems  to 
be  successful  and  achieve  the  benefits  of  using  commercial  items,  they  need  to  minimize 
the  uncertainties  and  manage  the  unique  risks  associated  with  commercial  items.  With 
any  risk,  awareness  of  lessons  learned  by  other  organization  that  have  implemented 
systems  using  commercial  items  will  help  build  or  strengthen  strategies  to  address  any 
unexpected  challenges  that  may  arise. 

The  next  chapter  presents  a  review  of  literature  and  lessons  learned  from 
programs  implementing  and  integrating  commercial  items  in  government  programs. 
These  sources  identify  a  consistent  pattern  of  unique  risks  associated  with  commercial 
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items.  Project  managers’  can  capitalize  on  these  many  “lesson  learned”  to  identify  and 
understand  the  risks  associated  with  commercial  items. 
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III.  IDENTIFICATION  OF  CHALLENGES  AND  RISKS  WITH 

COTS 


A.  INTRODUCTION 

Commercial  (COTS)-based  acquisition  strategy  can  be  viewed  as  a  risk 
management  approach  with  the  goal  of  reducing  or  eliminating  the  potentially  severe 
risks  and  resultant  adverse  effects  typical  of  custom-developed  systems.  However,  while 
the  use  of  commercial  products  can  help  to  deal  with  these  “custom  acquisition”  risks, 
using  commercial  products  also  introduces  other  forms  of  risk,  stemming  directly  from 
the  unique  characteristics  of  commercial  products.  This  increased  use  of  commercial 
products  by  government  organizations  is  creating  a  new  acquisition  operations  and 
support  environment  which  requires  that  a  standard  approach  be  established  for 
identifying  and  managing  (i.e.,  mitigating)  the  unique  risks  of  commercial  products.  This 
chapter  presents  many  of  the  unique  risk  factors  associated  with  developing  systems 
using  commercial  software.  These  factors  are  based  on  an  extensive  analysis  and  review 
of  common  government/industry  lessons-leamed  as  found  in  numerous  technical 
documents,  journals  and  other  literature  review.  Summaries  of  some  of  the  major 
documents  used  for  the  foundation  of  this  thesis  are  provided  in  this  chapter  with  the  full 
list  of  the  sources  provided  in  the  list  of  references. 

Examining  the  similarities  and  differences  of  organizations  that  have  applied 
commercial  (COTS)  products,  the  successes  and  failures  of  those  organizations,  will 
allow  us  to  identify  a  number  of  unique  factors  and  significant  capabilities  that  an 
organization  must  have  to  succeed  with  developing  a  system  using  commercial  items. 

B.  BACKGROUND  INFORMATION 

1.  United  States  Air  Force  Scientific  Advisory  Board:  Ensuring 

Successful  Implementation  Of  Commercial  Items  In  Air  Force  Systems, 

April  2000 

The  Air  Force  was  frustrated  over  the  lack  of  success  of  those  programs 
attempting  commercial  products  implementation  and  were  concerned  that  the  customers 
were  expecting  miracles.  Just  being  told  to  “maximize  use  of  commercial  (COTS)  items” 
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without  guidance,  training,  infrastructure,  processes,  tools,  metrics,  incentives,  and 
leadership  won’t  make  it  so.  A  panel  was  formed  to  look  into  a  broad  range  of 
commercial  hardware  and  software  products  involving  varying  degrees  of  integration  and 
complexity;  however,  the  commercial  hardware  considered  was  limited  to  computers  and 
electronics.  The  primary  purpose  of  this  report  was  to  capture  the  issues,  pitfalls,  myths, 
lessons  learned,  best  practices  and  critical  success  factors  associated  with  commercial 
(COTS)  items. 

The  panel  made  an  assessment  of  34  programs  and  organizations,  table  1, 
covering  three  broad  domains  -  management  information  systems  (MIS);  command, 
control,  communications  and  intelligence  (C3I);  and  weapon  systems  listing  the  well- 
recognized  benefits  and  several  not  so  well  recognized  risks  to  consider  when  utilizing 
commercial  items.  The  panel  observed  about  25  common  pitfalls  that  programs  are 
experiencing  with  most  struggling  with  the  technology,  processes  and  complexity  issues 
of  commercial  items  with  a  few  failing  miserably.  Most  of  these  pitfalls  could  have  been 
avoided  or  mitigated  if  appropriate  risk  management  processes  or  procedures  were  in 
place  that  people  understood  and  followed.  While  the  concept  of  a  commercial  (COTS)- 
based  system  is  easily  understood,  the  implementation  is  not.  The  successful 
implementation  of  commercial  products  impacts  virtually  every  aspect  of  the  acquisition 
process  including  acquisition  strategy,  source  selection,  program  management,  system 
development,  integration,  and  sustainment. 

Of  the  34  programs  interviewed,  five  were  considered  exemplary  and,  generally, 
those  with  the  most  experience  were  realizing  the  biggest  gains.  The  study  panel 
identified  the  common  characteristics  between  these  five  programs  and  strongly 
recommended  that  these  critical  success  factors  form  the  basis  of  an  implementation 
policy  within  the  Air  Force  (and  DOD).  To  serve  as  a  framework  to  drive  acquisition 
strategy,  source  selection,  program  management  and,  indirectly,  the  aerospace  industry. 
In  addition,  since  everyone  is  on  a  steep  learning  curve,  they  recommended  that  a 
periodic  or  annual  review  be  conducted  to  incorporate  additional  lessons  learned  into  the 
policy  until  the  situation  stabilizes. 
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Program/Organization 

Service 

Organization 

Advanced  Amphibious  Assault 

Vehicle 

USMC 

General  Dynamics  Amphibious 
Systems 

AF  Operational  Test  and  Evaluation 
Center 

USAF 

AFOTEC/CNR 

AFPEO/LI  for  Logistics  Info  SPO, 
Gunter,  AFB 

USAF 

AFPEO/LI 

AFRL  COTS  Initiatives  USAF 

USAF 

AFRL  COTS  Initiatives  USAF 
AFRL/MLM  &  /IFTA 

AWACS  Computer  Modernization 

USAF 

ESC/AWC 

B-2  Data  Storage 

USAF 

ASC/YSA 

B-2  EFX  99 

USAF 

Northrop  Grumman 

Boldstroke,  commonality  initiative 
Open  Systems  Architecture  & 

Software  Component  Technology 

The  Boeing  Company 

Bradley  Fighting  Vehicle 

USA 

United  Defense  LP 

CALCE  Electronic  Products  and 
Systems  Consortium 

University  of  Maryland 

DCAC/MRM  -  Define  &  Control 
Airplane  Configuration  / 
Manufacturing  Resource  Mgt 

The  Boeing  Company 

COTS  Supplier  Approaches 

DY  4  Systems 

Earth  Sensor 

TRW  Space  &  Technology 

Division 

F-117&F-119  Engine  Electronics 

USAF 

ASC/LPC  &  /LPR 

F-15E  COTS-based  Products  &  F-16 
Upgrade 

USAF 

ASC  &  ASC/YPV 

F-22  Program 

USAF 

ASC 

Global  Broadcast  System 

USAF 

Raytheon  Systems 

GPS,  Ground  Control  Segment 

USAF 

SMC/CZG 

GPS  Receiver 

USAF 

TRW  Space  &  Technology 

Division 

Ground  Station 

TRW  Space  &  Technology 

Division 

Reuse  of  COTS  Software 

GTE  Information  Systems 

Division 

JASPO,  Signal  Intelligence 
Infrastructure 

USAF 

ASC/RAJ 

Joint  Direct  Attack  Munitions 

USAF 

Program  Director 

Large  ADP  Systems  &  Software 
Development  Process 

TRW  Federal  Enterprise  Solutions 

Manufacturing  Resource  Planning 

USAF 

MRP  II  Program  Office 

COTS  Implementation  in  the 

Mobility  SPO 

USAF 

ASC  Commercial  Aircraft 

Program 
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Program/Organization 

Service 

Organization 

New  Attack  Submarine  and  Acoustic 
Rapid  COTS  Insertion  Programs 

USN 

Lockheed  Martin  Undersea 

Systems 

Enabling  E-Commerce  &  Distributed 
Computing 

Interoperability  Clearinghouse 

Office  of  the  Department  of  Defense 
Chief  Information  Officer 

OSD 

ASD/C3I  CIO 

PVS/EVS  -  Enterprise  Visibility 
Service 

Boeing  Information  Systems 

Deputy  Assistant  Secretary  for 
Management  Policy  &  Program 
Integration 

USAF 

SAF/AQX 

T-38C  Avionics  Upgrade  Program 

USAF 

ASC/EN 

T-6A  Joint  Primary  Aircraft  Training 
System 

USN 

USAF 

ASC/EN 

TacTech  (Parts  Management) 

Transition  Analysis  of  Component 
Technology,  Inc. 

Table  1.  34  Programs  or  Organizations  Reviewed  “From  [USAFOO]” 

2.  United  States  Air  Force  Space  Command:  Commercial  Space 

Opportunity  Study  (CSOS),  February  2000 

Caught  between  intensifying  warfighter  needs  in  space  and  tight  constraints  on  its 
budget,  the  Air  Force  has  been  encouraged  to  explore  options  in  the  commercial  market 
for  enhancing  space  capabilities  while  reducing  costs.  Beginning  in  1996,  the  Air  Force 
initiated  a  series  of  these  studies  to  determine  the  potential  of  commercial  space  to 
support  Air  Force  space  missions  and  requirements.  The  prior  studies  were  general  in 
nature,  but  identified  promising  opportunities  in  commercial  space  and  advised  Air  Force 
leaders  to  move  forward.  In  late  1997,  the  Chief  of  Staff,  USAF,  tasked  the  Chief 
Scientist  of  the  Air  Force  to  conduct  a  study  called  the  “Doable  Space”  Quick  Look 
Study.  This  study  found  that  the  military  potential  of  commercial  space  was  not  well 
defined  or  understood,  and  recommended  that  the  Air  Force  conduct  “an  aggressive  study 
on  exploiting  the  space  commercial  revolution.”  The  CSOS  was  chartered  to  build  on 
and  go  beyond  these  previous  studies  and  systematically  exposed  and  assessed  the 
technical,  operational,  policy  and  programmatic  implications  of  potential  commercial 
paths. 


18 


The  CSOS  set  out  to  identify  new  opportunities  to  satisfy  military  requirements 
while  reducing  Air  Force  costs,  expanding  capabilities  and/or  achieving  higher 
operational  efficiencies  focused  on  military  activities  that  the  commercial  market  was 
capable  of  implementing  in  both  near  and  long  term  opportunities.  The  objective  was  to 
develop  actions  to  address  the  issue  of  how  the  government  and  commercial  space 
community  can  best  work  together  to  capitalize  on  commercial  space  opportunities  and 
move  the  Air  Force  toward  an  integrated  architecture  that  best  serves  the  Air  Force’s 
needs  and  budget.  The  core  of  the  CSOS  approach  was  to  develop  business  cases 
through  intensive  discussions  with  industry,  and  to  show  whether  “commercialization” 
would  support  or  impair  national  security  and  readiness. 

The  study’s  approach  was  to  look  for  areas  where  interests  of  the  military  space 
community  and  the  commercial  space  community  coincide.  Proprietary  discussions  were 
held  with  interested  commercial  firms  in  areas  of:  customer  satisfaction,  market  share, 
product  development,  industry  growth  potential,  and  cost  control.  By  comparing  the  two 
sets  of  needs,  the  study  identified  common  areas  of  interest  to  both  communities.  These 
common  activities  were  launch,  command  and  control  (C2),  remote  sensing, 
communications,  and  navigation.  The  purpose  of  the  discussions  were  threefold: 

1)  To  ensure  that  industry  fully  understood  current  Air  Force  space  activities; 

2)  To  determine  whether  commercial  providers  could  and  would  provide 
equivalent  or  superior  services  at  costs  lower  than  government  costs;  and 

3)  To  find  out  whether  commercialization  could  be  executed  within  the  Air 
Force  and  not  violate  national  policy  requirements. 

Detailed  business  cases  were  solicited  from  those  firms  judged  to  have  the 
capability  to  provide  the  space  activity  or  function  in  question.  The  business  cases 
described  how  the  providers  would  meet  Air  Force  requirements  and  why  they  thought 
they  could  profitably  provide  the  services  or  products.  All  business  cases  were  reviewed 
through  several  iterations  until  the  study  leadership  felt  it  had  a  complete  understanding 
of  each  provider’s  concept  and  an  understanding  of  the  capability  of  the  relevant 
industrial  base  as  a  whole. 
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3.  Department  of  Defense  Inspector  General,  Lessons  Learned  from 
Acquisitions  of  Modified  Commercial  Items  and  Nondevelopmental  Items, 
Report  No.  97-219.  23  September  1997 

The  objective  of  this  report  was  to  determine  lessons  learned  from  the  acquisition 
management  of  Defense  systems  developed  and  procured  using  modified  commercial 
items  and  nondevelopmental  acquisition  strategies.  They  used  available  information 
from  ongoing  and  past  management  efforts  within  DOD  to  identify  many  lessons  learned 
from  acquiring  commercial  items.  They  also  reviewed  audit  reports  addressing 
acquisitions  of  modified  commercial  and  nondevelopmental  items  to  determine  whether 
the  acquisition  community  is  making  progress  in  developing  acquisition  strategies  that 
avoid  some  of  the  acquisition  difficulties  identified  in  earlier  audit  reports.  The  report 
summaries  and  categorizes  the  lesson  learned  from  the  buying  organizations  into  critical 
program  management  elements  and  evaluates  the  effectiveness  of  their  management 
controls. 

Specifically,  they  reviewed  37  programs,  table  2,  10  Army,  23  Navy,  and  4  Air 
Force  acquisition  programs  in  which  the  military  department  was  acquiring  modified 
commercial  and  nondevelopmental  items  as  entire  systems,  subsystems,  or  major 
components.  They  identified  91  lessons  learned  in  developing  acquisition  strategies  for 
program  definitions;  program  structures;  program  designs;  contracting;  program 
assessment;  and  decisions,  reviews,  and  periodic  reporting.  In  addition,  they  visited  22  of 
the  37  buying  organizations  to  discuss  specific  lesson  learned.  For  the  most  part,  the 
organizations  identified  program  uncertainties  involved  with  acquisitions  that  affect 
product  performance,  quality,  and  logistical  support.  The  report  then  identifies  key 
acquisition  strategies  that  would  be  disseminate  to  provide  buying  organizations  with 
useful  infonnation  on  how  to  acquire  modified  commercial  and  nondevelopmental  items 
from  commercial  suppliers. 


Department  of  the  Army 

Modified 

Commercial 

Nondevelopmental 

Biological  Integrated  Detection 

System 

X 

Cargo  Utility  Commercial  Vehicle 

X 

Cargo  Utility  Global  Positioning 
System  Receiver 

X 
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Department  of  the  Army 

Modified 

Commercial 

Nondevelopmental 

Communications-Electronics 

Command  Commercial 
Communications  Technology  Lab 

X 

Deployable  Universal  Combat 
Earthmover 

X 

Lightweight  Multiband  Satellite 
Terminal 

X 

Lightweight  Video  Reconnaissance 
System 

X 

National  Automotive  Center 

X 

X 

Near  Term  Digital  Radio 

X 

Precision  Lightweight  Global 
Positioning  System  Receiver 

X 

DEPARTMENT  OF  THE 

Modified 

Nondevelopmental 

NAVY 

Commercial 

Advanced  Deployable  System 

X 

ARC-210  Very  High  Frequency/Ultra 
High  Frequency  Radio 

X 

Battle  Group  Passive  Horizon 
Extension  Surface  Terminal 

X 

Combat  Systems  Engineering 

X 

X 

Common  Support  Equipment 

X 

Control  Display  Navigation  Unit 

X 

Fixed  Distributed  System 

X 

Ground  Proximity  Warning  System 

X 

X 

High  Frequency  Radio  Group 

X 

Hull,  Mechanical  and  Electrical 
Equipment  Data  Resources  System 

X 

X 

Joint  Maritime  Command  Information 
System-Afloat 

X 

Joint  Power  Projection/Real  Time 
Support 

X 

Medium  Tactical  Vehicle 
Remanufacturing 

X 

Miniature  Digital  Assigned  Multiple 
Access 

X 

New  Attack  Submarine 

X 

PI 00  Portable  Firefighting  Group 

X 

Riverine  Assault  Craft 

X 

Strategic  Systems  Program 

X 

X 

Submarine  Message  Buffer 

X 

21 


Department  of  the  Navy 

Modified 

Commercial 

Nondevelopmental 

Surface  Ship  Torpedo  Defense 

a.  Launched  Expendable 
Acoustic  Device 

X 

b.  Multi-Sensor  Torpedo 
Recognition  and  Alertment  Processor 

X 

Surveillance  Towed  Array  Sensor 
System 

X 

Surveillance  Towed  Array  Sensor 
System-  Low  Frequency  Active 

X 

Department  of  the  Air  Force 

Modified 

Commercial 

Nondevelopmental 

C-130J  Aircraft 

X 

Commercial  Aircraft  Program 

X 

Military  Products  From  Commercial 
Lines 

X 

T-1A  Aircraft 

X 

Table  2.  37  Modified  Commercial  and  Nondevelopmental  Programs  Reviewed  From 

“[DODI97]” 

4.  FAA,  COTS  Risk  Mitigation  Guide:  Practical  Methods  For  Effective 

COTS  Acquisition  and  Life  Cycle  Support,  June  2002 

Since  the  introduction  of  the  Federal  Aviation  Administration’s  (FAA) 
Acquisition  Management  System  (AMS)  in  1996,  the  agency  has  fielded  numerous 
commercial  (COTS)-based  systems  into  the  National  Airspace  System  (NAS).  However, 
due  the  lack  of  any  available  internal  or  external  guidance  on  how  to  manage  the  unique 
risks  associated  with  commercial  item  acquisitions,  the  FAA  as  well  as  many  other 
Government  agencies  has  had  a  variety  of  experiences,  many  of  them  adding  to  system 
cost,  schedule  and  perfonnance  risks.  This  guide  was  established  to  capitalize  on  these 
many  “lessons-leamed”  from  government  and  industry  and  imbed  them  in  a  practical 
manner  within  the  context  of  an  acquisition  management  process  to  more  effectively 
acquire  and  provide  life  cycle  support  for  commercial  (COTS)-based  systems.  The  guide 
provides  the  necessary  underlying  structured  for  a  standardize  approach  to  identify 
commercial  (COTS)  risks,  analyzes  the  likelihood  and  consequences  of  the  risks  and 
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determine  appropriate  mitigation  strategies  to  minimize  their  impact.  It  uses  a  set  of 
worksheets  and  schedules  to 

•  Collect  the  information 

•  Assess  the  risk  level 

•  Select  the  most  suitable  solution 

•  Develop  and  deploy  the  solutions 

•  Develop  the  technical  rational  to  justify  funding  requirements 

Using  the  programmatic  risk  management  element  of  it  systems  engineering 
progress  (Figure  3.1),  the  FAA  collected  information  on  commonly  experienced 
government  and  industry  “lessons-leamed”  in  the  areas  of  reliability,  maintainability, 
availability,  supportability  trends,  and  market  research  activities.  They  used  the  Internet, 
in-house  experience,  plus  commercial  and  DOD  publications  and  then  converted  them 
into  risk  factors.  The  risk  factors  were  subsequently  analyzed  to  determine  practical  risk 
mitigation  strategies  that  could  be  included  as  part  of  early  program  acquisition,  planning 
and  which  could  be  continuously  applied  throughout  a  system’s  lifecycle.  The 
information  contained  in  this  COTS  risk  mitigation  guide  would  be  incorporated  as  part 
of  the  overall  risk  management  program  for  both  existing  and  new  start  commercial 
(COTS)-based  system  acquisitions.  Because  technology  and  commercial  products  evolve 
rapidly,  this  process  must  be  updated  on  a  frequent  (proactive)  basis  to  avoid  disruption 
(reactive)  of  system  operations  helping  to  avoid  the  disruptions  caused  by  commercial 
items. 
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Figure  2.  FAA  Programmatic  Risk  Management  Process  “From  [FAAC02]” 

5.  Department  of  Defense,  Office  of  the  Under  Secretary  of  Defense  for 

Acquisition  and  Technology,  SD-2:  Buying  Commercial  & 

Nondevelopmental  Items:  A  Handbook,  April  1996 

The  Department  of  Defense  must  leam  to  use  commercial  and  nondevelopmental 
items  (NDI)  effectively.  Our  ability  to  field  affordable,  state-of-the-art  systems  when  they 
are  needed,  and  to  buy  the  millions  of  items  needed  to  support  our  troops  and  fielded 
systems,  depends  on  efficient  use  of  available  resources.  The  use  of  commercial  items  is 
no  longer  a  question  of  “yes  or  no”  but  a  question  of  to  what  degree.  This  handbook  was 
developed  as  a  guide  for  acquisition  managers  and  personnel  in  other  functional  areas 
who  are  involved  in  buying  commercial  and  nondevelopmental  items  (NDI).  It  is 
intended  to  help  these  individuals  buy  these  items  without  inhibiting  their  use  of  creative 
and  innovative  strategies. 

It  offers  guidance  on  commercial  and  NDI  acquisitions.  It  addresses  the  entire 
spectrum  of  acquisitions  from  systems,  subsystems,  assemblies,  parts,  and  items  of 
supply.  This  guide  does  not  present  a  “cookbook”  approach  to  commercial  and  NDI 
acquisitions;  however,  it  does  offer  best  practices  and  “lessons  learned”  along  with 
additional  “things  to  consider”,  when  buying  commercial  items.  It  presents  case  studies 
to  demonstrate  the  effectiveness  of  using  commercial  products  and  practices  in: 
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•  Defining  Requirements 

•  Acquisition  Process 

•  Logistics  Support 

•  Test  and  Evaluation 

•  Product  Assurance 

With  relatively  short  lead  times  for  fielding  commercial  and  nondevelopmental 
items,  acquisition  decision  must  not  be  made  until  tradeoff  factors  are  identified, 
analyzed,  and  compared  with  other  alternatives.  In  determining  if  use  of  a  commercial  or 
nondevelopmental  item  is  feasible,  personnel  must  tailor  the  guidance  provided  to  the 
circumstances  of  their  particular  acquisitions  and  devote  more  program  resources  to 
addressing  life-cycle  support  as  more  of  the  quantifiable  program  risk  areas  become 
known. 

6.  Naval  Sea  Systems  Command,  Commercial  Off-The-Shelf  and  Non- 

Developmental  Items  Handbook,  March  2000 

This  document  was  created  in  response  to  the  challenge  of  DOD  to  use  more 
commercial  products  in  its  military  systems  and  the  feedback  from  the  Naval  Sea 
Systems  Command  (NAVSEA)  Commercial-Off-the-Shelf  (COTS)  Workshop  held  in 
Norfolk,  VA  in  August  1998,  where  the  Fleet  and  the  NAVSEA  user  community 
expressed  the  need  for  NAVSEA  guidance  in  the  utilization  of  commercial  items.  Given 
the  fiscal  constraints  under  which  NAVSEA  operates,  it  has  significantly  increased 
Commercial  Off  The  Shelf/Non-Developmental  Item  (COTS/NDI)  in  system 
acquisitions.  Employing  COTS/NDI  is  a  prudent  means  of  lowering  the  costs  of 
acquiring  equipment  and  systems  that  satisfy  the  Navy's  needs.  However,  effective 
management  of  commercial  hardware  and  software  in  Navy  systems  presents  difficult 
and  different  challenges  than  traditional  item  acquisition  and  life  cycle  support. 

The  document  provides  overall  guidance  and  outlines  approaches  for  developing 
successful  acquisition,  integration,  and  maintenance  support  strategies  whenever 
commercial  products  are  employed  in  military  applications  under  the  cognizance  of  the 
Naval  Sea  Systems  Command.  It  is  written  from  a  global  perspective  and  is  intended  to 
will  help  managers  and  implementers  decide  “what”  factors  to  consider  when  employing, 

integrating  and  supporting  COTS/NDI  into  systems;  providing  the  framework  to  develop, 
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manage  and  execute  a  comprehensive,  cost  effective,  COTS/NDI  program  based  on  DOD 
policy.  By  consolidating  points  from  previously  prepared  plans,  reports  and  studies,  they 
leverage  industry  experience,  lessons  learned  in  other  military  applications,  and  current 
“best  practices”  to  allow  individuals  to  tailor  these  “what”  factors  to  their  specific 
program. 

It  focuses  on  the  acquisition  and  life  cycle  support  of  COTS/NDI  for  hardware 
and  software  for  all  NAVSEA  programs  (excluding  Nuclear  Propulsion  Programs  under 
the  cognizance  of  SEA  08).  It  provides  guidance  for  those  disciplines  involved  in  all 
phases  of  the  COTS/NDI  acquisition  and  life  cycle  support  process: 

•  Program  Management 

•  Technology  Management 

•  Systems  Engineering 

•  Test  and  Evaluation 

•  Configuration  Management 

•  Logistics  Support 

•  Product  Assurance 

While  it  is  understood  that  every  acquisition  projects  is  unique  and  will  vary 
greatly  in  complexity  and  requirements,  the  guiding  tenets  contained  within  this 
handbook  should  be  reviewed,  addressed  and  tailored  accordingly  to  ensure  successful 
application  of  COTS/NDI  products  to  mission  and  program  needs.  The  considerations 
contained  in  this  guidance  are  not  a  “cookbook”  for  the  application  of  COTS/NDI  in 
NAVSEA  Systems  but  intended  to  provoke  questions  that  can  then  be  answered  by 
obtaining  additional  information  on  the  COTS/NDI  product. 

7.  Department  of  Defense,  Data  &  Analysis  Center  for  Software, 

Commercial-Off-The-Shelf  (COTS):  A  Survey,  December  2000 

The  goal  of  this  report  was  to  survey  the  state  of  the  practice  in  commercial 
(COTS)-based  development  and  provide  evidences  of  both  successful  use  and  failures  of 
commercial  (COTS)  items  in  projects  using  them.  Each  commercial  software  component 
used  is  less  code  that  needs  to  be  designed  and  implemented  by  the  developers. 
However,  the  developer  is  faced  with  the  problem  of  ensuring  that  the  commercial 
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product  does  perform  the  functionality  that  it  claims  to  perform,  that  it  does  not 
intentionally  perfonn  functionality  to  be  harmful  to  the  system,  that  it  will  not  adversely 
affect  the  system  and  that  it  can  robustly  respond  to  failures  and  anomalous  inputs  to 
prevent  errors  from  propagating  through  the  entire  system. 

This  report  discusses  the  definition  of  commercial  items  and  commercial  (COTS)- 
based  system,  listing  the  pros,  cons  and  issues  in  commercial  (COTS)-based 
development.  The  use  of  commercial  products  in  software  development  can  require  a 
considerable  integration  effort.  Early  estimation  of  this  effort  will  help  developers  to 
choose  the  right  commercial  products  and  to  decide  whether  to  develop  their  own 
software  instead  of  using  a  commercial  item. 

The  central  part  of  this  report  is  dedicated  to  survey  methods  and  techniques  that 
can  be  useful  in  commercial  (COTS)-based  development.  There  are  little  or  no 
techniques  that  allow  a  user  to  assess  the  dependability  of  a  commercial  item  (in  the 
sense  of  availability  of  the  functionality  promised  by  the  documentation,  reliability  of  the 
functionality,  availability  and  quality  of  documentation);  therefore,  successful 
implementation  of  commercial  items  depends  on  several  factors.  They  try  to  summarize 
and  analyze  these  factors  by  discussing  how  these  factors  influence  the  success  of  a 
project  using  commercial  items.  They  then  propose  a  process  to  support  commercial 
(COTS)-based  development  with  emerging  standards  and  techniques  for  component 
integration  discussed.  By  aggregating  the  factors  they  argue  that  using  a  dependable 
commercial  item  in  a  non-critical  project  based  on  a  single  commercial  item  is  probably  a 
reasonable  choice.  On  the  other  hand,  integrating  several  commercial  products,  just 
released,  in  a  highly  critical  project  means  probably  asking  for  trouble.  In  between  lies  a 
twilight  zone  where  the  decision  on  using  commercial  items  has  to  be  carefully  evaluated 
case  by  case. 


8.  A  Management  Guide  to  Software  Maintenance  in  COTS-Based 
Systems,  November  1998 

The  use  of  commercial  products  can  significantly  change  the  process  by  which 
systems  are  maintained  in  their  operational  phase.  We  are  just  beginning  to  experience 
and  understand  these  changes,  and  to  recognize  that  life-cycle  planning  for  commercial 
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(COTS)-based  systems  must  take  into  account,  in  early  planning,  the  issues  that  must  be 
confronted  in  order  to  facilitate  the  maintenance  phase.  The  objective  of  this  guidebook 
was  twofold: 

1)  To  provide  planning  guidance  that  results  in  low  risk  and  cost-effective 
strategies  for  maintaining  Commercial  Off-the-Shelf  (COTS)  software 
products  in  commercial  (COTS)-based  systems,  and 

2)  To  provide  guidance  on  the  preparation  of  a  commercial  (COTS)  Software 
Life-Cycle  Management  Plan. 

The  authors  believe  there  is  no  single  way  to  manage  and  sustain  all  commercial 
(COTS)-based  systems;  therefore,  they  developed  a  generic  commercial  (COTS)  life- 
cycle  plan  and  guidelines,  describing  the  changes  in  the  software  maintenance  process  for 
systems  using  commercial  products  and  how  this  process  can  be  tailored  for  each  system 
depending  on  its  specific  requirements  and  constraints.  The  basis  for  this  guidance  was  a 
review  of  DOD  and  industry  experiences  and  lessons  learned  in  commercial  (COTS) 
product  applications,  and  attendance  at  meetings/workshops  whose  focus  was  on  the 
implementation  of  commercial  (COTS)-based  systems.  The  guidebook  considers  the 
issues  and  risks  in  using  commercial  software  over  the  life  cycle  and  how  to  control 
them.  Each  commercial  (COTS)-based  system  must  look  at  and  control  the  risks 
associated  with  the  operational  and  technical  characteristics  of  the  system,  the 
administrative  policies  and  constraints  placed  on  the  program,  and  its  financial  situation. 
With  the  use  of  commercial  products,  the  operation  and  maintenance  phase  starts  sooner 
and  continues  for  a  much  longer  portion  of  a  system’s  life  cycle.  This  makes  early  life 
cycle  planning  for  maintenance  of  commercial  products  even  more  important. 

C.  RISKS  AND  CHALLENGES 

Those  who  have  followed  the  commercial  (COTS)  path  have  been  learning  the 
hard  way  that  “just  buying  commercial  items”  does  not  necessarily  lead  to  all  of  the 
desired  benefits;  problems  and  difficulties  in  acquisition  procedures,  product 
development,  and  logistics  support,  as  well  as  the  skills  and  experience  required  of 
personnel  supporting  the  project  are  also  introduced  by  the  use  of  commercial  products. 
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The  extensive  review  of  common  government/industry  lessons-leamed  and  other 
literature  identified  the  need  for  identifying  and  understanding  the  unique  risks  factors  (or 
characteristics)  and  challenges  associated  with  using  commercial  products.  Risks  that  are 
not  identified  have  the  greatest  chance  of  becoming  problems,  while  risks  that  are 
identified  and  assessed  have  the  greatest  chance  of  being  resolved.  This  section  proposes 
a  set  of  commercial  items  risk  factors  and  challenges  that  can  be  classified  into  three 
categories  of  commercial  risks  with  each  category  being  further  divided  into  elements  or 
attributes.  These  categories  are  based  on  the  numerous  technical  documents  and  personal 
experience.  These  categories  are: 

•  Process:  The  key  considerations  for  developing  and  executing  a 
successful  acquisition  process  with  the  system/program  requirements 
driving  the  organization  to  consider  a  commercial  solution  and  the  “fit”  of 
those  requirements  with  available  commercial  application  package(s). 
Key  areas  are  organizational,  planning,  tracking,  contractual  parameters, 
and  evaluation  of  vendors’  experience  and  past  performance. 

•  Technology:  The  technical  “fit”  of  the  commercial  product(s)  with  the 
existing  and  planned  technical  architecture,  which  supports  an 
organization.  How  well  the  selected  product  will  perform  in  the 
environment  provided  by  the  system.  This  includes  the  organization’s 
inherent  technical  challenges,  such  as  the  number  and  complexity  of 
interfaces. 

•  Implementation/Logistics  Support:  The  process  contains  intermediate 
and  final  work  product  characteristics  for  the  delivery  of  a  commercial 
solution  within  an  organization  that  includes  -  but  is  not  limited  to 
performance  measures,  vendors  availability  of  support,  testing  and 
managing  organizational  change. 

1.  Process  Risk  Factor:  Commercial  Standards 

Military  equipment  is  required  to  operate  under  conditions  not  always  required  of 
commercial  equipment.  For  instance:  gunfire  vibration,  hot  and  cold  extremes,  and 
nuclear  hardness  are  normal  operating  environments  for  military  equipment.  Commercial 
elements  must  be  selected  with  these  requirements  in  mind  and  there  may  not  always  be  a 
commercial  element  that  will  work. 

Until  recently  the  government  drove  technology  development  for  military 
applications  with  large  infusions  of  research  and  development  (R&D)  funding  for 
custom-developed  systems.  The  government  could  afford  to  specify  exactly  what  was 
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desired,  set  requirements,  and  therefore  promoted  a  “buyers”  market  of  firms  interested  in 
meeting  this  demand.  However,  commercial  products  are  typically  designed  and  built  to 
a  variety  of  commercial  standards  that  provide  high-level  guidance  on  such  product 
characteristics  as  perfonnance,  quality  and  inter-operability  fostering  a  “sellers” 
marketplace  that  is  no  longer  driven  by  government  R&D  but  by  a  much  larger  (and  more 
profitable)  commercial  customer-base.  This  means  that  the  commercial  vendors  have 
several  customers  and  their  products  are  manufactured  to  meet  more  general  consumer 
demands,  instead  of  being  configured  to  meet  specific  and  often-inflexible  government 
requirements. 

This  competitive  environment  and  the  rapid  advances  in  the  underlying 
technologies  both  drive  and  allow  commercial  product  manufacturers  to  anticipate 
customer  demands  and  to  quickly  develop  and  market  their  commercial  products. 
Leaving  the  government  with  no  control  and  minimal  influence  over  how  the  commercial 
product  evolves.  Hence,  if  operational  requirements  are  viewed  as  not  negotiable,  and 
the  suppliers  are  unwilling  to  modify  their  commercial  products  to  meet  a  unique  military 
need,  then  the  probability  of  finding  an  exact  match  between  requirement  and 
commercial  product  is  diminished.  [USAFOO] 

2.  Process  Risk  Factor:  License  Agreements 

Licensing  is  the  vehicle  for  securing  the  use  of  products  that  you  need;  data  rights 
and  warranties  are  marketplace  vehicles  for  protecting  you  (and  the  vendors)  in  the  long¬ 
term  use  of  those  products.  Understanding,  mastering,  and  negotiating  the  licensing 
agreements  with  the  vendor  can  have  a  tremendous  impact  on  the  success  of  your 
program.  The  fee  structures  of  licenses  and  maintenance  services  may  change  without 
warning  potentially  resulting  in  a  large  cost  impact.  Different  licensing  and  maintenance 
support  options  are  available  and  negotiable  which  are  sometimes  unknown  to  customers. 
Enterprise  licensing  is  rarely  available  and  not  many  users  within  the  DOD  are  using  the 
same  commercial  software  for  the  same  purpose. 
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3. 


Process  Risk  Factor:  Vendors  Past  Performance 


While  many  individual  commercial  products  from  different  manufacturers  might 
satisfy  a  particular  set  of  functional  requirements,  there  can  be  marked  differences  from 
one  product  to  the  next.  Differences  in  the  components  manufacturers  choose  to  use, 
quality  assurance  practices,  manufacturing  processes,  labor  force  composition,  market 
share,  product  support,  upward/downward  compatibility,  corporate  longevity,  etc.  can  all 
affect  the  quality  and  therefore  desirability  of  the  commercial  products  that  are  offered 
for  sale.  The  “buyer  beware”  maxim  applies  when  choosing  among  apparently  similar 
products.  A  vendor  with  a  limited  product  line  is  likely  to  sacrifice  a  product  to 
compensate  for  adverse  market  financial  flux,  while  a  vendor  that  employs  ad-hoc 
development  practices  may  not  be  able  to  sustain  long-term  product  evolution,  with  other 
vendors  offering  little  or  no  warning  for  produce  releases/upgrades  forcing  the  maintainer 
into  reactive  evolution  mode  to  deal  with  obsolescence  issues. 

4.  Technology  Risk  Factor:  Rapid  and  Asynchronous  Changes 

Rapid  turnover  in  the  commercial  product  can  be  both  a  risk  and  a  missed 
opportunity  for  the  program  manager  who  is  unaware  of  these  changes.  If  the  sole 
objective  of  a  system  upgrade  is  to  capture  new  technology  more  cheaply,  then  the  use  of 
commercial  products  may  suffice.  But  many  DOD  systems  have  a  30-  to  50-year  lifetime 
or  more,  while  the  average  commercial  component  is  upgraded  every  6  to  12  months  and 
new  technology  emerges  about  every  18  to  24  months.  Changes  to  a  commercial  product 
is  driven  primarily  by  the  vendors’  perceptions  of  how  to  achieve  a  greater  market  share, 
how  to  anticipate  customer  demands  and  to  quickly  develop  and  market  their  commercial 
product  to  meet  these  demands.  Thus  the  changes  are  based  on  what  will  sell  well  to  the 
largest  number  of  current  and  potential  users,  not  on  the  unique  needs  of  your  particular 
programs.  Vendors  can  add  or  take  away  functionality  and  may  not  place  the  same 
priority  as  you  do  on  a  change  that  you  need  or  the  retention  of  a  feature  on  which  you 
rely.  The  money  that  is  saved  by  using  a  commercial  product  with  proprietary  interfaces 
can  quickly  be  lost  in  maintenance  as  products  and  their  interfaces  change  with  the 
marketplace  forcing  customers  to  accept  the  upgrades  of  the  new  product  in  order  to 

obtain  the  desired  functionality.  Even  if  the  expected  lifetime  of  a  system  is  only  five  to 
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ten  years,  the  fluctuations  in  commercial  products  and  technology  result  in  a  state  of 
constant  change  for  any  system  employing  them. 

Program  management  generally  cannot  control  the  frequency  or  the  content  of 
new  commercial  releases.  Vendors  are  continually  producing  new  products  as  well  as 
revising  the  products  that  are  already  on  the  market  with  the  timing  of  a  new  commercial 
product  release  tending  to  be  asynchronous  and  independent  of  the  new  releases  of  other 
commercial  products  and  components  in  the  system.  Upgrading  to  the  latest  version  can 
result  in  risks  such  as  the  following  and  stresses  the  need  to  fully  test  each  upgrade  before 
incorporating  it  into  the  system: 

•  The  new  software  version  is  incompatible  with  other  commercial  software 
products  in  the  system,  necessitating  updating  of  those  products  too. 

•  The  new  version  has  new  data  fonnats  that  require  changes  to  be  made  to 
the  formats  and  contents  of  existing  files  and  databases  that  were  created 
by  prior  versions  of  the  commercial  software. 

•  The  new  version  of  the  software  is  incompatible  with  the  version  of  the 
hardware  that  is  in  the  system. 

•  A  new  version  of  the  hardware  is  introduced  into  the  system  that  forces 
changes  to  the  existing  versions  of  the  software  to  make  them  compatible 
or  because  timing  has  changed  under  the  new  hardware. 

•  The  new  version  of  the  software  changes  the  user  interface  in  ways  that 
require  retraining  operators. 

•  Changes  in  the  consumption  of  time  or  memory  resources  by  upgrades  to 
commercial  software  are  not  compatible  with  the  system  requirements  or 
the  hardware  capacities. 

•  The  new  version  forces  changes  in  the  operational  capabilities  of  the 
system  because  it  no  longer  supports  those  capabilities  in  the  same  way  or 
at  all. 

•  A  new  version  may  provide  more  capabilities  that  may  have  to  be 
suppressed  or  restricted  due  to  security  concerns. 

5.  Technology  Risk  Factor:  Integration 

The  complexity  of  commercial  software  interfaces  (e.g.  operating  system)  with 
other  commercial  software  products/applications,  middleware,  glue  code,  custom/legacy 
interfaces  and  integrating  these  multiple  commercial  products  within  one  single  system 
can  lead  to  many  interoperability  problems: 
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•  Lack  of  commonality  with  other  products.  It  is  possible  (in  fact,  likely) 
that  using  a  closed  commercial  product  commits  the  user  to  proprietary 
interfaces  and  solutions  that  are  not  common  with  any  other  product, 
component,  or  system. 

•  Perfonnance  feature  clash.  Commercial  software  vendors  typically 
overload  their  systems  and  includes  more  features  and  functionality  than 
are  normally  needed.  Precautions  must  be  taken  during  system 
development  and  subsequent  upgrades  to  assure  that  these  unused  features 
do  not  clash  with  other  software  products.  System’s  architects  must 
provide  a  way  of  masking  the  unwanted  functionality  so  that  it  is 
inaccessible  to  the  end-users  and  system  programmers. 

•  Multiple  configurations.  Changing  generations  of  commercial  products 
will  occur.  Depending  on  system  complexity,  the  number  of  systems  to  be 
fielded  and  the  length  of  time  it  takes  to  deploy  them,  the  number  of 
configurations  could  be  significant.  It  is  not  uncommon  for  part  numbers 
and  software  release  identifiers  to  be  the  same  but  have  different  features 
or  contents.  For  example,  one  production  lot  can  be  functionally 
equivalent  to  the  next  lot  but  contain  different  components  and 
subassemblies.  If  a  product  contains  firmware  or  if  it  is  a  software 
product,  revisions  can  be  made  to  subsequent  product  releases  to  correct 
deficiencies  or  to  add  unique  features  to  enhance  product  marketability.  A 
commercial  product  manufacturer  may  or  may  not  elect  to  identify  these 
configuration  changes  to  its  customers. 


6.  Technology  Risk  Factor:  Reliability 

Today,  with  the  rush  to  bring  many  products  to  market,  commercial  products  are 
notoriously  error  prone.  One  must  recognize  that  a  new  product  in  a  hot  market  segment 
will  have  problems,  some  potentially  crippling  to  a  system's  reliability.  This  is  a  major 
concern  since  all  military  equipment  must  be  highly  reliable  in  the  field.  This  sometimes 
requires  equipment  with  failure  rates  not  achievable  with  available  commercial  elements, 
forcing  us  to  have  an  understanding  about  the  vendor’s  track  record  in  building  reliable 
commercial  products.  For  certain  applications,  occasional  errors  and  downtime  may  be 
acceptable.  For  other  applications,  the  requirements  may  specify  a  Mean  Time  Between 
Failures  and  Mean  Time  To  Repair  that  are  very  demanding-resulting  in  higher  project 
cost. 


Evaluating  commercial  product  reliability  is  somewhat  different  than  evaluating 
the  reliability  of  new  development  products.  With  commercial  items  the  basic  product  is 
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already  designed  and  its  reliability  established;  however,  detailed  engineering  and 
manufacturing  data  for  commercial  products  is  frequently  not  available.  The 
Government  is  not  involved  in  the  design  process  and  production  testing  for  a 
commercial  product.  So  the  Government  cannot  continuously  evaluate  the  reliability 
during  design  reviews,  through  analysis,  or  based  on  production  test  results. 

Consequently,  the  reliability  assessment  should  be  an  operational  assessment  of 
the  military  application  in  the  expected  military  environments  since  the  buyer  cannot 
control  the  basic  design  of  a  commercial  item.  The  commercial  product  must  pass  the 
same  reliability  evaluations  as  the  host  components;  otherwise  the  commercial  product 
will  be  the  weakest  link  in  the  chain  of  components  and  will  be  the  detenninant  of 
software  system  reliability.  The  essential  reliability  analysis/tasks  that  must  be 
performed  are  reliability  predictions,  system  level  Failure  Mode  Analysis,  Failure 
Reporting  and  Tracking  Analysis,  and  reliability  verification.  Consider  the  following 
when  evaluating  and  fielding  commercial  products: 

•  Reliability  predictions  may  be  difficult  to  obtain  from  the  vendor. 

•  Lack  of  data  may  limit  the  depth  of  failure  mode  analysis  that  can  be  done 
on  commercial  products. 

•  Vendor’s  definition  of  reliability  data  may  be  different  than 

DOD/Government  standards  (e.g.,  Ao,  MTBF,  MTTR). 

7.  Technology  Risk  Factor:  Information  Security 

When  the  government  develops  its  own  custom  systems,  it  can  specify  and 
develop  system  security  characteristics  very  precisely.  Although  vendors  provide 
products  with  built  in  security  features  that  address  the  commercial  components 
interoperability  issues,  these  products  are  typically  developed  to  commercial  standards 
with  insecure  defaults  that  introduces  potentially  significant  security  risks  for  several 
reasons. 

•  The  increased  inter-operability  among  different  products  that  meet 

commercial  standards  raises  the  chances  that  unauthorized  access  can  be 
gained. 

•  The  use  of  commercial  standards  allows  a  greater  number  of  people  to  be 
familiar  with  the  software  protocols  used  to  manage  information.  This 
knowledge  can  be  used  to  access  or  disrupt  information  flow.  The  “open- 
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ness”  of  a  particular  architecture,  the  degree  to  which  it  links  with  other 
external  commercial  (COTS)-based  systems,  and  the  nature  of  the  security 
measures  in  place  will  determine  the  extent  to  which  the  products  and 
systems  using  them  are  susceptible  to  unauthorized  access.  [FAAC02] 

Most  commercial  software  is  developed  and  implemented  outside  this  country.  It 
is  common  in  software  product  development  in  the  United  States  to  use  teams  from  other 
countries.  In  a  government  project  with  particular  security  requirements  this  presents 
major  risk  factors  that  may  be  unacceptable.  Because  a  commercial  product  is  essentially 
a  black  box,  in  the  sense  that  the  implementation  of  the  software  is  most  often  hidden, 
leading  to  the  possibility  that  a  trap  door  or  “Trojan  Horse”  may  be  embedded  in  the 
code,  there  may  be  a  backdoor  feature  in  the  code,  or  unexpected  capabilities.  As  a 
result,  specific  security  relating  to  project  requirements  may  not  be  guaranteed,  as  the 
security  of  the  commercial  products  implementation  cannot  be  ascertained. 

8.  Implementation/Logistical  Risk  Factor:  Product  Obsolescence 
(discontinuation) 

Commercial  products  life  cycles  are  generally  much  shorter  with  new  versions  of 
a  commercial  software  package  appearing  as  frequently  as  every  18  months.  As 
succeeding  generations  of  commercial  products  are  introduced  into  the  commercial 
market,  the  manufacturer  must  determine  at  what  point  when  it  is  no  longer  profitable  or 
desirable  to  support  the  older  generation  products.  The  manufacturer  must  make  a 
tradeoff  between  selling  its  newer  product  line  while  at  the  same  time  not  alienating  the 
older  generation  product  consumer  base.  After  three  or  four  upgrades,  the  manufacturer 
may  choose  to  no  longer  maintain  the  earlier  version  incorporated  in  the  military  system. 
Also,  a  commercial  product  may  be  selected  for  a  particular  niche  feature.  If  it  turns  out 
that  the  commercial  market  is  not  interested  in  this  capability,  there  could  be  a  lack  of 
support  or  subsequent  revisions  that  may  exclude  the  feature  entirely. 

When  a  commercial  product  is  projected  to  be  nearing  end-of-life  (EOL)  (i.e.,  out 
of  production)  or  end-of-service  (EOS)  (i.e.,  no  longer  supported  by  the  manufacturer), 
the  effects  of  these  projected  changes  of  state  on  the  product  and  on  systems  using  the 
product  must  be  examined  to  detennine  what  action  if  any  is  needed.  It  is  not  a  foregone 
conclusion  that  all  products  declared  to  be  EOL  or  EOS  need  to  be  replaced  immediately 
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by  newer  versions  of  those  products.  Effects  can  range  from  no  impact  to  high  impact. 
The  obsolescence  support  options  that  are  available  to  address  these  impacts  can  range 
from  taking  no  action  to  making  a  major  system  redesign  [FAAC02].  The  categories  of 
impacts  due  to  obsolescence  are  defined  as  follows: 

•  No  impact  -  In  this  case  the  product’s  projected  End  of  Life/End  of 
Service  status  has  no  impact  on  the  product  or  on  any  system  using  that 
product  and  therefore  requires  no  action.  The  commercial  product  is 
considered  reliable  and  there  are  sufficient  spares  (at  acceptable  prices, 
within  the  market  or  on-hand)  to  support  the  projected  failure-driven 
demand  over  a  pre-determined  timeframe. 

•  Low  impact  -  This  situation  typically  requires  compatibility  testing  for  the 
new  product  and  a  documentation  change  to  identify  the  new  product  as  a 
suitable  alternative  replacement  part  upon  failure  of  the  old  part.  The 
manufacturer’s  next  generation  product  is  compatible  i.e., 
interchangeable);  if  there  are  other  manufacturer  products  that  are 
compatible;  and  if  there  are  no  associated  changes  to  interfacing  products 
within  the  system. 

•  Medium  impact  -  This  category  of  impact,  like  the  low  impact  category, 
also  applies  when  a  commercial  product  must  eventually  be  replaced.  The 
manufacturer’s  next  generation  product  requires  minor  software  changes 
and/or  if  related  changes  to  interfacing  products  are  required. 

•  High  impact  -  This  category  of  impact,  like  both  the  low  and  medium 
impact  categories,  applies  when  a  commercial  product  must  eventually  be 
replaced.  However,  a  major  impact  situation  exists  if  there  are  no 
compatible  replacement  products  or  technologies  available  on  the  market. 
This  situation  typically  calls  for  a  major  redesign  or  an  integrated  system 
change. 

9.  Implementation/Logistical  Risk  Factor:  Proprietary  Data 

A  major  drawback  of  including  commercial  items  in  a  software  system  is  the  lack 
of  visibility  into  how  the  commercial  components  were  developed  and  an  incomplete 
understanding  of  the  components’  behavioral  properties  [SCHNOO].  A  commercial 
product  manufacturer  remains  in  business  because  it  owns  and  controls  the  research  and 
manufacturing  processes  needed  to  meet  market  demands  and  to  keep  product  costs 
competitive.  As  consumers,  we  have  little  insight  into  the  specifics  of  how  a  commercial 
product  has  been  developed  and  at  times  even  into  the  details  of  how  it  behaves  and  why. 
This  lack  of  visibility  can  hamper  efforts  to  integrate  the  product  with  others  to  create  a 
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larger  system  since  the  vendor  may  not  be  willing  to  provide  detailed  interface  design 
documentation.  As  a  result,  the  commercial  product  must  be  viewed  as  a  “black  box” 
with  defined  interface  and  perfonnance  characteristics  but  allowing  no  insight  into  the 
internal  composition  of  that  product.  Because  we  do  not  have  access  to  the  source  code, 
developers  cannot  modify  the  code  to  change  the  functionality  of  the  commercial  product 
(perhaps  this  is  a  good  thing!). 


10.  Implementation/Logistical  Risk  Factor:  Underestimated  Costs 

Accelerating  the  introduction  of  commercial  products  into  government  and 
military  systems  has  been  advertised  as  a  “faster,  better,  cheaper”  way  of  meeting 
requirements.  However,  unless  a  risk  management  program  includes  proactive  mitigation 
strategies  looking  at  the  total  ownership  of  cost  for  commercial  products,  the  initial  cost 
benefits  can  be  offset  by  the  often  more  costly  fixes  of  the  risks  that  weren’t  effectively 
managed.  Examples  of  the  cost  considerations  for  a  commercial  (COTS)-based 
acquisition  strategy  that  need  to  be  included  as  part  of  a  total  cost  of  ownership  analysis 
include  [FAAC02]: 

•  Inadequate-planning  costs  -  Probably  the  major  life  cycle  cost-driver 
associated  with  the  use  of  commercial  products  is  the  lack  of  effective 
planning  and  budgeting.  When  a  program  fails  to  apply  commercial  risk 
mitigation  strategies,  the  program  then  loses  the  advantage  of  proactive 
planning  and  becomes  increasingly  reactive  to  emerging  commercial- 
driven  obsolescence  situations. 

•  Test  and  integration  costs  -Programs  often  underestimate  the  impact  of 
testing  commercial  items.  In  addition  to  the  actual  costs  of  the  test 
facilities  needed  to  support  the  possibility  of  multiple  system 
configurations,  different  commercial  products  with  varying  characteristics 
typically  require  that  “glue  code”  be  developed  to  allow  the  products  to 
interact  effectively.  Each  product  must  be  tested  for  compliance  to 
performance  requirements,  conformance  to  open  system  standards  and 
compatibility  with  the  system  with  which  it  will  be  integrated. 

•  Modification  costs  -  In  some  cases  a  commercial  product  must  be 
modified  to  meet  a  particular  or  unique  requirement.  There  is  a  cost  to 
actually  modify  the  commercial  product  itself.  There  is  also  a  cost  to 
assume  life  cycle  management  responsibility  for  that  specific  product 
because  modifying  a  commercial  product  typically  voids  (unless  functions 
are  incorporated  as  part  of  the  commercial  product  line)  any  warranty  and 
the  vendor  will  no  longer  provide  support.  This  forces  the  life  cycle 
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support  for  that  product  to  be  the  responsibility  of  the  acquiring  activity. 
Costs  for  documentation,  maintenance,  training  and  spares  costs  will 
increase  in  this  situation  and  must  be  planned  for  in  the  life  cycle 
budgeting  for  that  modified  product.  In  addition,  regression  testing  at  the 
system  level  may  be  needed  to  ensure  that  the  modification  does  not 
change  the  expected  perfonnance  of  the  system. 

•  Configuration  management  costs  -  A  consequence  of  using  rapidly 
changing  commercial  products  within  a  given  system  is  the  strong 
likelihood  that  an  acquisition  of  multiple  copies  of  that  system  will  include 
more  than  one  configuration  of  the  commercial  products  used  in  the 
system.  This  situation  not  only  demands  a  rigorous  application  of 
configuration  management  (CM)  processes  to  document  and  manage 
system  baselines  but  also  requires  that  test  facilities  can  replicate  all 
fielded  configuration  baselines. 

•  Continuous  system  engineering  costs  -Commercial  products  forces  a 
continuous  system  engineering  effort,  which  adds  additional  cost  to  a 
program.  Because  commercial  (COTS)-based  systems  are  dynamic  in 
nature,  continuous  systems  engineering  activities  are  needed  to  perform 
market  surveillance/research/investigation;  analyze  obsolescence 
projections;  detennine  the  available  options  to  limit  obsolescence  impacts; 
and  integrate  the  resulting  information  with  new  requirements  and  field 
data  as  part  of  the  overall  integrated  program  planning.  You  need  to  be 
able  to  go  out  and  see  what  is  going  on  in  the  marketplace  and  do  some 
technology  forecasting. 

•  Obsolescence  management  costs  -  There  are  costs  associated  with  vendors 
or  suppliers  either  dropping  support  for  a  commercial  item  or  going  out  of 
business.  The  continuous  system  engineering  activities  needed  to  manage 
obsolescence  can  result  in  more  frequent  engineering  changes  to  the 
system.  The  development,  deployment  and  configuration  management  of 
these  changes  is  an  added  cost  that  must  be  included  in  all  commercial 
(COTS)-based  system  program  planning.  These  costs  must  continuously 
be  refined  as  system  product  obsolescence  information  is  gathered  and 
analyzed. 


11.  Implementation/Logistical  Risk  Factor:  Testing 

Commercial  item’s  capabilities  may  not  always  be  as  stated  and  demand 
excessive  testing.  There  may  be  “hidden  behaviors”  associated  with  the  commercial 
item,  or  bugs  that  affect  the  system.  System  integration  and  system  level  testing 
becoming  even  more  vital,  particularly  in  commercial  (COTS)-based  systems  with  many 
components  (commercial  (COTS),  NDI,  custom,  legacy)  where  interoperability  issues 
abound;  therefore,  one  must  ensure  that  the  system  fulfills  all  specified  requirements  and 
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that  catastrophic  faults  are  detected  early.  When  commercial  products  are  modified,  the 
system  may  exhibit  behavior  different  from  the  baseline  requiring  additional  testing.  In 
the  past,  unique  custom  designs  were  static  so  that  the  system  tested  was  the  system  to  be 
manufactured  and  deployed.  Because  of  the  volatility  of  systems  incorporating 
commercial  products,  the  system  that  is  subjected  to  initial  operational,  testing  and 
evaluation  (IOT&E)  is  likely  to  be  different  from  the  system  that  enters  production,  since 
upgrades  will  have  been  incorporated.  This  constant  evolution  of  a  system  is  a  cause  of, 
since  the  consequences  of  the  changes  introduced  into  the  initial  production  design  are 
unknown. 

Testing  and  fault  isolations  are  further  complicated  by  the  reality  of  restricted 
visibility  into  the  behavior  of  the  commercial  product  with  any  documentation  that  you 
may  have,  often  incomplete  and  inconsistent,  causing  you  to  shift  from  a  “white  box”  to 
“black  box”  testing  process  [SCHNOO],  The  integrator/testers  must  thoroughly  test  all 
inputs  and  outputs  to  “prove”  to  a  vendor  with  potentially  significant  evidence  that  a 
particular  commercial  component  is  failing  in  a  particular  way,  whether  a  detected  failure 
is  in  a  single  component  (and  then,  which  one)  or  in  the  interactions  among  two  or  more 
components.  This  may  take  significant  resources  (time  and  very  skilled  technical  staff)  to 
isolate  and  resolve  with  the  vendors.  It  is  not  the  vendor’s  responsibility  for  the  ultimate 
success  of  your  system;  rather  it  becomes  the  integrating  organization’s  responsibility. 
You  might  want  them  to  fix  some  bug  or  something,  and  they  may  or  may  not  do  it  in 
that  given  release.  Sit  down  in  advance  with  your  vendors  to  detennine  the  routine  for 
working  out  solutions  to  problems  that  will  be  encountered  and  find  a  way  to 
cooperatively  work  together  to  find  a  satisfactory  resolution. 

D.  SUMMARY 

This  chapter  summarizes  some  of  the  major  government/industry  lessons-learned 
and  other  technical  infonnation  to  form  a  foundation  for  understanding  and  identifying 
the  unique  risk  factors  associated  with  developing  systems  using  commercial  software.  It 
proposed  a  set  of  commercial  item  risk  factors  and  challenges  that  were  classified  into 
three  categories:  process,  technology,  and  implementation/support,  with  each  category 

being  further  divided  into  specific  elements  or  attributes.  Only  by  understanding  and 
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addressing  the  unique  factors  imposed  by  commercial  items  will  program  managers  be 
able  to  attain  their  benefits  and  move  towards  market-oriented  business  practices  that  are 
better  suited  to  the  acquisition  and  life  cycle  support  of  commercial  (COTS)-based 
systems. 
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IV.  COTS  RISK  QUESTIONNAIRE/CHECKLIST 


A.  INTRODUCTION 

Widespread  use  of  commercial  products  in  complex  software  systems  poses  many 
unique  challenges  and  risks  to  both  the  developers  and  managers  of  systems  using 
commercial  items.  It  is  virtually  impossible  to  develop,  modify  or  purchase  commercial 
software  without  incurring  risks.  These  risks  can  be  known,  unknown,  or  unknowable. 
Known  risks  are  those  that  one  or  more  project  personnel  are  aware  as  concerns.  The 
unknown  risks  are  those  that  would  surface  (i.e.,  become  known)  if  project  personnel 
were  given  the  right  opportunity,  cues,  and  information.  The  unknowable  risks  are  those 
that,  even  in  principle,  none  could  foresee.  Hence  these  risks,  while  potentially  critical  to 
project  success,  are  beyond  the  purview  of  any  risk  identification  method. 

Identifying  the  unique  risk  factors,  assessing  these  factors,  and  controlling  them 
are  the  keys  to  proper  risk  management  with  commercial  items.  Identification  surfaces 
risks  before  they  become  problems  and  adversely  affect  a  project.  The  sooner  risks  are 
identified,  the  better  off  the  software  managers,  system  engineers,  project  manager  or 
decision-managers  will  be  able  to  monitor,  adequately  mitigate,  and  resolve  the  risks; 
thus,  achieving  the  desired  benefits  of  using  commercial  items  by  ensuring  a  rapid  and 
successful  delivery  of  the  system. 

B.  IDENTIFICATION  OF  RISKS 

This  section  presents  a  questionnaire  (Appendix  B),  checklist,  for  adopting  and 
integrating  commercial  product(s)  into  systems  or  programs.  The  questionnaire  is 
intended  to  provide  a  guideline  (reminders),  to  any  of  the  participants  on  the  program, 
whether  on  the  acquiring  side  or  the  contracting  side,  and  focus  their  attention  on  the 
possible  risk  factors  that  individuals  need  to  understand  when  using  commercial  items.  It 
consists  of  two  sections  and  is  designed  to  provide  a  systematical  tool,  starting  point,  for 
managers  to  identify,  investigate,  and  plan  for  the  unique  risks  associated  with 
implementing  commercial  items.  It  incorporates  some  of  the  most  significant  lessons 
learned  from  a  variety  of  commercial  implementations  [ITRB99]  and  helps  you  evaluate 
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the  risk  by  determining  the  severity  of  these  risks  for  your  own  organization.  While  it 
may  not  be  complete  and  has  some  obvious  weaknesses,  the  checklist  provides  an 
organization,  which  is  considering  the  adoption  and  integration  of  commercial  items, 
insight  into  the  areas  that  must  be  carefully  considered.  It  also  serves  an  important 
purpose  for  large-scale  organizations  by  providing  a  repeatable  approach  for  commercial 
items  risk  evaluation. 


1.  Section  I.  Demographic  Information. 

Collecting  background  information  about  the  organization  and  experience  of  the 
individual(s),  with  commercial  product(s),  completing  the  questionnaire. 


2.  Section  II.  Risk  Questions 

A  modification  was  done  to  the  Information  Technology  Resources  Board’s 
(ITRB)  “Risk  Profile”[ITRB99],  in  order  to  structure  42  questions  around  the  three  broad 
categories  that  represents  critical  aspects  required  for  the  successful  implementation  of 
commercial  items  as  identified  in  chapter  three:  process,  technology,  and 

implementation/logistics  support,  defined  below,  with  several  questions  for  each 
category.  Even  though  some  questions  may  not  pertain  to  every  project,  these  questions 
can  be  modified  accordingly  to  meet  the  needs  of  the  software  project.  Each  question 
prompts  you,  the  respondent,  to  think  about  key  factors  for  a  successful  commercial 
product(s)  implementation  and  how  these  factors  pertain  to  your  project  within  your  own 
organization. 

•  Process:  The  key  considerations  for  developing  and  executing  a 
successful  acquisition  process  with  the  system/program  requirements 
driving  the  organization  to  consider  a  commercial  solution  and  the  “fit”  of 
those  requirements  with  available  commercial  product(s).  Key  areas  are 
organizational,  planning,  tracking,  contractual  parameters,  and  evaluation 
of  vendor’s  experience  and  past  performance. 

•  Technology:  The  technical  “fit”  of  the  commercial  product(s)  with  the 
existing  and  planned  technical  architecture,  which  supports  an 
organization.  This  includes  the  organization’s  inherent  technical 
challenges,  such  as  the  number  and  complexity  of  interfaces. 
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•  Implementation/Logistics  Support:  The  process  contains  intermediate 
and  final  work  product  characteristics  for  the  delivery  of  a  commercial 
solution  within  an  organization  that  includes  -  but  is  not  limited  to 
performance  measures,  vendors  availability  of  support,  testing  and 
managing  organizational  change. 


C.  ASSESSING  RESULTS 

Completing  the  questions  and  assessing/compiling  the  results  should  help 
managers  better  understand  the  significance  level  of  risk  associated  with  implementing  a 
commercial  product(s)  and  assist  in  identifying  their  causes  given  current  business  needs 
and  organizational  conditions.  In  turn,  this  knowledge  will  help  guide  the  managers  and 
let  them  take  the  steps  necessary  to  minimize  specific  risks  associated  with  the 
implementation  of  a  commercial  product(s)  and  formulate  a  strategy  for  acquiring 
commercial  product(s)  for  their  organization. 


1.  Risk  Severity  Rating 

Answers  to  each  question  are  provided  by  the  choice  a,  b  or  c,  which  correlate  to 
the  three  levels  of  risk:  low,  medium  and  high,  respectively  with  points  assigned  for 
different  levels.  The  level  of  risk  is  somewhat  subjective  and  should  be  based  on  the 
experienced  judgment  of  your  best  technical  people  with  assigned  responsibility. 
However,  user  input  and  feedback,  along  with  industry  comments  also  need  to  be 
considered. 

•  Low  risk,  point  value  =  1,  Actions  within  the  scope  of  the  planned 
program  and  normal  management  attention  should  result  in  maintaining  an 
acceptable  level  of  risk. 

•  Moderate  risk,  point  value  =2,  Corrective  actions  and/or  careful 
monitoring  of  status  by  management  are  required  to  reduce  risk  or  to  see 
that  the  level  of  risk  does  not  increase. 

•  High  risk,  point  value  =  3,  Significant  corrective  action  and  high  priority 
management  attention  are  required  to  achieve  an  acceptable  level  of  risk. 
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2.  Calculating  the  Risks 

To  arrive  at  the  programs  total  risk,  each  individual  section  of  the  risk  categories 
must  be  examined.  A  box  is  provided  for  adding  the  total  number  of  a,  b,  or  c  responses 
for  each  section.  The  table  below  illustrates  this  concept.  The  first  column  is  the 
categories  of  the  risks  associated  with  a  program  implementing  commercial  product(s): 
process,  technology,  and  implementation/logistics  support.  Moving  to  the  right  across 
the  other  columns  in  the  table  you  will  find  space  for  recording  the  number  of  a,  b,  and  c 
responses  associated  with  each  risk  category.  These  individual  point  values  are  then 
summed  to  provide  the  total  risk  severity  rating,  point  score,  indicated  in  the  lower  right 
comer  of  the  table.  This  determines  where  the  point  total  falls  on  the  scale  shown  and 
identifies  the  programs  overall  risk  rating  of  using  commercial  items.  In  turn,  each 
individual  category  risk  rating  could  be  determined  by  calculating  the  responses  for  that 
individual  category,  based  on  the  number  of  questions  for  that  category. 

If  most  of  your  responses  were  a's,  your  organization  has  a  low  risk  profile  for 
successfully  implementing  a  commercial  application  package(s).  While  an  overall  low 
risk  is  a  strong  indicator,  it  is  important  to  note  that  this  does  not  mean  there  is  "no-risk". 
Every  commercial  product(s)  implementation  involves  some  degree  of  risk. 

If  most  of  your  responses  were  b's,  your  organization  has  a  moderate  risk  for 
implementing  a  commercial  product(s).  Carefully  examine  the  questions,  particularly 
with  medium  risks  (b)  and  high  risks  (c)  responses  to  identify  specific  vulnerabilities  and 
record  them  in  a  database  or  risk  mitigation  plan  with  action  items  or  task  plans  to  track 
risks  to  closure. 

If  most  of  your  responses  were  c's,  your  organization  has  a  high  degree  of  risk  for 
implementing  a  commercial  product(s).  Review  the  questions  to  help  your  organization 
identify  critical  areas  that  need  to  be  reexamined  regardless  of  its  commercial 
implementation  phase. 
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Total  Calculations 


b’s 


a’s 


c’s 


Totals  from  Process:  — 

Totals  from  Technology:  _ 

Totals  from  Implementation/  _  _ 

Support 

Totals  ]  ^ 

Program  Totals 

Total  a’s _ x  (1) 

+  Total  b’s _ x  (2) 

+  Total  c’s _ x  (3) 

Project  Total  = 


MODERATE 


LOW 


123 


82 
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Table  3.  Commercial  Item  Risk  Profile 


D.  SUMMARY 

This  chapter  described  a  method  for  facilitating  the  systematic  and  repeatable 
identification  of  risks  associated  with  use  of  commercial  items.  It  presents  a 
questionnaire,  checklist,  to  start  the  managers  and  engineers  thinking  about  and  planning 
on  how  to  avoid,  mitigate  and  accept  the  risks  inherent  in  any  software  project  using 
commercial  items.  Risks  that  are  not  identified  have  the  greatest  chance  of  becoming 
problems.  Many  organizations  who  attempt  to  implement  a  commercial  products(s) 
without  sufficient  analysis  and  preparation  encounter  significant  challenges  that  can  be 
related  to  the  business  processes  used  to  build  systems,  technologies  used  to  construct  the 
system,  and  logistical  support  issues  that  inevitably  arise.  As  a  minimum,  the  project 
manager,  software  manager,  system  engineer/manager,  any  software  technical  leads,  and 
the  software  engineers,  should  fill  out  and  discuss  the  risk  identification  method  stated  in 

this  chapter.  Careful  consideration  of  these  issues  will  help  to  minimize  your 
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organization's  risk  severity  rating  and  curb  future  expenditures.  With  any  level  of  risk, 
awareness  of  lessons  learned  by  other  organizations  that  have  implemented  commercial 
items  will  help  build  or  strengthen  strategies  to  address  any  unexpected  challenges  that 
may  arise. 
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V.  COTS  RISK  QUESTIONNAIRE  ANALYSIS 


A.  INTRODUCTION 

The  questionnaire  was  sent  electronically  to  the  Office  of  the  Secretary  of 
Defense,  Command  Control  and  Communication  (C3I)  Commercial  Policies  and 
Oversight  Office,  the  Army’s  Communication  and  Electronics  Command  (CECOM),  the 
Marines  Systems  Command  along  with  numerous  other  individual  project  offices  that 
utilize  commercial  products  to  elicit  responses,  capture  experiences,  and  record  the 
results  of  the  questionnaire  for  active  Department  of  Defense  (DOD)  projects  that  utilize 
commercial  products.  The  intent  was  for  the  major  commands  or  organizations  to 
distribute  the  questionnaire  to  project  offices  within  their  organizations  that  use 
commercial  products  to  complete,  revealing  the  highest  risks  for  their  projects.  The 
following  are  responses  from  active  programs  using  commercial  product(s). 

B.  DEFENSE  LOGISTIC  AGENCY  (DLA):  BUSINESS  SYSTEMS 

MODERNIZATION 

The  Defense  Logistic  Agency  (DLA)  is  the  primary  logistics  provider  for  the 
DOD  and  is  continually  seeking  ways  to  improve  and  reduce  the  cost  of  distribution.  It  is 
undergoing  a  major  Information  Technology  and  reengineering  transformation,  Business 
Systems  Modernization  (BSM),  to  modernize  the  agency’s  business  practices  by  using 
best  DLA  and  commercial  practices  and  commercial  software;  thus,  allowing  them  to 
rely  on  industry  for  support  and  reduce  inventory  levels  by  hundreds  of  millions  of 
dollars. 

The  new  information  technology  system  being  implemented  allows  DLA  to 
exploit  new  emerging  technologies  and  streamline  its  supply  chain  process  by 
consolidating  its  operations  to  one  level  of  national  inventory,  generating  great 
economies  of  scale  as  well  as  total  visibility  of  all  DOD  stocks.  Letting  them  achieve  the 
proven  benefits  of  commercial  software,  provide  better  service  at  higher  workloads, 
reduce  the  cost,  and  pass  the  savings  of  this  improved  process  back  to  their  military 
customers. 
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The  questionnaire  was  completed  by  an  individual  with  over  10  years  of 
experience  on  building  systems  using  commercial  products;  however,  his  experience  was 
with  minor  projects,  not  one  of  this  magnitude  (enterprise-wide).  Some  of  the  lessons 
learned  from  acquiring  and  developing  commercial  product(s)  for  this  project  are: 

•  Do  not  modify  core  COTS  software 

•  Willingness  to  adapt 

•  Completeness  of  requirements 


Table  4  provides  the  risk  profile  for  DLA’s  BSM  project  with  the  following 
factors  being  identified  as  a  high  risk.  The  complete  results  from  the  questionnaire  are 
contained  in  Appendix  C. 

•  Many  functions  supported  by  the  commercial  product 

•  Very  complex  interfaces  between  commercial  product  and  other  systems 

•  Many  of  the  interfaces  must  remain  unchanged 

•  Extensive  training  required  to  operate  and  maintain  the  commercial 
product 


Total  Calculations 


Totals  from  Process: 

Totals  from  Technology: 

Totals  from  Implementation/ 
Support 

Totals 


a’s 

b’s 

c’s 

17 

6 

0 

5 

3 

4 

5 

5 

1 

22 

14 

5 

Program  Totals 

Total  a’s  22  x  (1) 

+  Total  b’s  14  x  (2) 

+  Total  c’s  5  x  (3) 

Project  Total  =  65 


MODERATE 


LOW 


123 


82 


3 . 1 . 1 
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Table  4.  DLA  BSM  Commercial  Item  Risk  Profile 
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C.  ARMY  HUMAN  RESOURCE  SYSTEM  (AHRS) 

The  Army  Human  Resource  System  Product  Management  Office  provides  and 
maintains  the  personal  management  information  system  for  the  active  Army,  the  Army 
Human  Resource  System  (AHRS)  Super  Server.  It  is  an  integrated  automated  field 
military  personnel  management  system  designed  to: 

•  Serve  America's  Army  during  mobilization,  war,  and  demobilization 

•  Serve  the  Active  Anny  during  peacetime 

•  Provide  commanders  a  responsive  personnel  management  system,  which 
facilitates  peacetime  personnel  strength  accounting  management  and 
wartime  operations. 

The  questionnaire  was  completed  by  an  individual  with  over  30  years  of 
experience  on  building  at  least  20  systems  using  commercial  products.  Some  of  the 
lessons  learned  from  acquiring  and  developing  commercial  product(s)  for  this  project  are: 

•  Do  not  use  software  that  does  not  have  a  long  standing  commercial  user 
base 

•  Do  not  allow  GOTS  products  to  be  forced  onto  your  program.  These  are 
generally  built  with  commercial  products  no  longer  in  business 

Table  5  provides  the  risk  profile  for  the  Army’s  Human  Resource  System  project 
with  the  following  high  risks  factors  being  identified.  The  complete  results  from  the 
questionnaire  are  contained  in  Appendix  D. 

•  Many  functions  supported  by  the  commercial  product 

•  Much  customization  of  the  commercial  product  needed  to  meet  the  needs 
of  the  organization 

•  Testing  approach  was  designed  for  traditional  testing  of  a  system 
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Total  Calculations 


Totals  from  Process: 

Totals  from  Technology: 
Totals  from  Implementation/ 
Support 


Totals 


a’s 

b’s 

c’s 

17 

l 

0 

7 

3 

2 

6 

4 

1 

30 

7 

3 

Program  Totals 

Total  a’s  30  x  (1) 

+  Total  b’s  7  x  (2) 

+  Total  c’s  3  x  (3) 

Project  Total  =  53 
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1 . 1 
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1  J 

Table  5.  Army’s  Human  Resource  System  (AHRS)  Commercial  Item  Risk  Profile 

D.  ARMY  (CECOM)  COMMUNICATIONS  SOFTWARE  ENGINEERING 
SUPPORT  DIVISION  (CSES) 

The  Communications  Software  Engineering  Support  (CSES)  Division  provides 
life  cycle  software  engineering  services  to  the  Program  Executive  Office  for  Command, 
Control,  and  Communications  Systems  (PEO  C3S),  as  well  as  other  Department  of  the 
Army  and  DOD  organizations  and  agencies.  These  services  include  all  activities 
necessary  to  ensure  the  reliability,  maintainability,  interoperability,  and  configuration 
integrity  of  the  software  components  used  in  communications  and  related  Mission 
Critical  Defense  Systems  (MCDSs)  for  both  systems  under  development  and  systems 
deployed  to  operational  units  worldwide  assuring  joint  interoperability  of  tactical 
switching  and  network  management  software/firmware  through  quality  assurance  testing; 
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oversight  and  certification  recommendation  of  all  new  software  releases;  supporting 
acquisition  of  interoperable  tactical  switching  and  network  management  systems;  and 
serving  as  the  single  point  of  contact  for  the  warfighters  in  all  matters  involving  switch 
interoperability. 

The  questionnaire  was  completed  by  an  individual  with  four  years  of  experience 
making  recommendations  on  commercial  product(s)  that  were  no  longer  supported  or 
reached  their  end  of  life  with  two  recommendations  for  commercial  replacements  being 
integrated.  Some  of  the  lessons  learned  from  acquiring  and  developing  commercial 
product(s)  for  this  project  are: 

•  Known  your  requirements  well 

•  Assess  and  evaluate  different  available  commercial  products  based  on  the 
requirements  well  in  advance 

•  Close,  continuous,  and  active  partnership  among  the  vendor,  customers, 
developer,  and  most  importantly  the  users 

Table  6  provides  the  risk  profile  for  the  Army’s  Human  Resource  System  project 
with  the  following  high  risks  factors  being  identified.  The  complete  results  from  the 
questionnaire  are  contained  in  Appendix  E. 

•  Vendor  unknown  or  poor  performance  in  integrating  the  commercial 
application 

•  Vendor  has  a  track  record  of  exceeding  total  life  cycle  cost  estimates 

•  No  discussion  with  the  vendor  for  future  plans  of  the  commercial  product 

•  System  is  a  new  system 

•  Very  complex  interfaces  between  commercial  product  and  other  systems 

•  No  flexibility  in  the  design  to  allow  future  changes  in  functionally 

•  Program  has  not  gather  information  from  other  organizations  that 
implement  commercial  applications 

•  No  performance  measures  to  measure  the  effectiveness  of  the  commercial 
product 

•  No  contingency  plan  for  commercial  vendor  going  out  of  business 

•  Extensive  training  required  to  operate  and  maintain  the  commercial 
product 
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Total  Calculations 


Totals  from  Process: 

Totals  from  Technology: 
Totals  from  Implementation/ 
Support 


Totals 


a’s 

b’s 

c’s 

3 

12 

3 

2 

7 

3 

0 

7 

4 

5 

26 

10 

Program  Totals 

Total  a’s  5  x  (1) 

+  Total  b’s  26  x  (2) 

+  Total  c’s  10  x  (3) 

Project  Total  =  87 


123  82  41 


Table  6.  Army’s  Communications  Software  Engineering  Support  Division 

Commercial  Item  Risk  Profile 

E.  ARMY  GLOBAL  COMBAT  SUPPORT  SYSTEM  (GCSS) 

Global  Combat  Support  System  (GCSS)-ARMY  is  the  largest  and  most  complex 
information  technology  program  in  the  Anny,  which  will,  over  time,  replace  or  interface 
to  all  of  our  existing  CSS  automated  systems  and  provide  automatic,  user-transparent 
communications  for  routine  transactions.  It  will  encompass  personnel,  financial,  medical, 
and  other  non-logistics  CSS  functions  and  be  made  up  of  a  series  of  functional  modules 
(or  Product  Lines)  such  as  Supply,  Property,  Maintenance  and  Management  with  each 
module  operating  within  the  Defense  Information  Infrastructure  and  run  at  any  level  or 
organization  where  the  Army  performs  that  function. 

Designing  the  modules  at  the  same  time  gives  the  modules  a  common  look  and 
feel  using  a  graphical  user  interface  with  point  and  click  techniques  with  Commercial- 
Off-The  Shelf  hardware  and  the  Windows  NT  operating  systems  used  to  support  the  new 
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software.  These  two  features  coupled  with  embedded  training  will  simplify  initial  and 
sustainment  training  requirements  tremendously. 

The  questionnaire  was  completed  by  an  individual  with  one  year  of  experience  on 
building  systems  using  commercial  products  with  no  lessons  learned  and  since  the 
program  is  under  implementation  with  over  130,000  expected  users  with  multiple  and 
different  training  requirement;  there,  he  could  not  answer  questions  like  “how  efficient  is 
it?” 


Table  7  provides  the  risk  profile  for  the  Army’s  Global  Command  Support 
System  project.  Even  though  the  project  has  a  low  profile  for  successful  implementation 
with  commercial  products,  it  is  important  to  note  that  this  profile  does  not  mean  a  "no- 
risk"  profile.  Every  commercial  product(s)  implementation  involves  some  degree  of  risk. 
The  complete  results  from  the  questionnaire  are  contained  in  Appendix  F. 

Total  Calculations 

Totals  from  Process: 

Totals  from  Technology: 

Totals  from  Implementation/ 

Support 

Totals 


Program  Totals 

Total  a’s  39  x  (1) 

+  Total  b’s  1  x  (2) 

+  Total  c’s  0  x  (3) 

Project  Total  =  41 


a’s 

b’s 

c’s 

17 

o 

0 

12 

0 

0 

10 

1 

0 

39 

1 

0 

MODERATE 


LOW 


I  I  1 


123 


82 


41 


Table  7.  Army’s  Global  Combat  Support  System  Commercial  Item  Risk  Profile 
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F.  MARINE  CORPS  COMBAT  VEHICLE  TRAINING  SIMULATOR 

The  Marines  want  to  be  able  to  use  the  simulators  to  cover  the  full  breadth  of 
training  from  the  beginning  driver  all  the  way  to  the  tactical  driver,  who  is  in  severe  off¬ 
road  conditions  and  inclement  weather  and  blackout  conditions.  Too  often,  driving 
simulations  are  based  on  flight  simulations  or  video  games,  and  therefore  are 
unsatisfactory  for  serious  training.  The  simulator  will  not  only  display  exterior  driving 
conditions,  but  also  will  provide  a  realistic  environment  of  the  interior  of  the  vehicle. 
Everything  the  driver  will  have  available  to  him  in  the  vehicle  will  be  available  to  him  in 
the  simulator. 

A  Semi-Automated  Forces  (SAF)  demonstration  system  will  be  developed  that 
will  be  networked  with  Raydon-developed  LAV-25  and  M1A1  Abrams  Tank  simulators 
via  the  High  Level  Architecture  (HLA)  protocol  be  and  operate  in  two  basic  modes.  As 
an  exercise  editor,  it  is  used  to  define  entities  and  their  characteristics,  build  a  scenario 
containing  a  selection  of  those  entities  and  assign  appropriate  behavior  to  those  entities. 
As  a  runtime  engine  and  Situation  Awareness  Display,  the  SAF  controls  the  behavior  of 
its  entities  and  displays  the  composite  worldview  of  all  entities  in  the  simulation, 
including  those  external  to  the  SAF.  There  will  also  be  three  visual  databases:  a  desert 
database,  a  geotypical  European  database  and  a  geotypical  rural  database  to  support 
training  in  both  Visual  and  Thennal  modes. 

The  questionnaire  was  completed  by  an  individual  with  five  years  of  experience 
building  systems  with  commercial  products;  however,  she  has  never  participated  in 
selecting  commercial  software  for  the  integration  into  a  system.  Table  8  provides  the  risk 
profile  for  the  Marine  Corps’  Ground  Transportation  Engineer  Systems  project  with  the 
following  high  risks  factors  being  identified.  The  complete  results  from  the  questionnaire 
are  contained  in  Appendix  G. 

•  No  data  right  negotiated  into  the  contract 

•  Uncertain  about  what  licensing  agreements  are  needed 

•  Much  customization  of  the  commercial  product  needed  to  meet  the  needs 
of  the  organization 

•  Many  functions  supported  by  the  commercial  product 
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•  Program  has  not  gather  information  from  other  organizations  that 
implement  commercial  applications 

•  No  performance  measures  to  measure  the  impact  and  effectiveness  of  the 
commercial  product 

•  Extensive  training  required  to  operate  and  maintain  the  commercial 
product 

•  Very  little  training  resources  available  to  the  customer 

•  Testing  approach  was  designed  for  traditional  testing  of  a  system 


Total  Calculations 

a’s 

b’s 

c’s 

Totals  from  Process: 

5 

li 

2 

Totals  from  Technology: 

5 

5 

2 

Totals  from  Implementation/ 

Support 

2 

4 

5 

Totals 

12 

20 

9 

Program  Totals 

Total  a’s  12  x  (1) 

+  Total  b’s  20_  x  (2) 

+  Total  c’s  9  x  (3) 

Project  Total  =  79 


123  82  41 

Table  8.  Marine  Corps  Combat  Vehicle  Training  Vehicle  Simulator  Commercial  Item 

Risk  Profile 

G.  ARMY  COMMON  SOFTWARE  PROGRAM 

The  Army's  Common  Software  Program  is  based  upon  the  Global  Command  and 
Control  System  (GCCS)  which  has  two  main  objectives:  the  replacement  of  the  World- 
Wide  Command  and  Control  System  (WWMCCS)  and  the  implementation  of  the 
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Command,  Control,  Communications,  Computers,  and  Intelligence  (C4I).  For  the 
Warrior.  GCCS  is  designed  to  become  the  single,  global  command,  control, 
communications,  and  intelligence  system  to  support  the  war  fighter,  whether  from  a 
foxhole  or  from  a  command  center.  The  GCCS  system  is  based  upon  the  Common 
Operating  Environment  (COE)  which  provides  the  infrastructure  for  all  command  and 
control  systems.  This  COE  consists  of  an  integrated  architecture  made  up  of  hardware 
and  software  that  provides  standard,  modular,  system  support  and  applications  support 
software  for  a  set  of  functional  applications.  The  COE  software  is  a  multi-layered  open 
system  architecture  consisting  of  modular  functional  applications,  application  support 
software,  standard  system  support  software  which  is  designed  to  operate  on  a  standard 
suite  of  computers  and  consists  of  19  functional  areas.  It  fully  supports  a  reuse  program 
that  is  domain  specific,  architecture  centric,  and  systematic,  implementing  the 
Department  of  Defense  (DOD)  software  reuse  vision  and  strategy. 

The  questionnaire  was  completed  by  an  individual  with  two  years  of  experience 
building  systems  with  commercial  products  and  has  participated  only  once  in  the 
selection  of  software  components  that  were  later  adapted  or  integrated.  Some  of  the 
lessons  learned  from  acquiring  and  developing  commercial  product(s)  for  this  project  are: 

•  Never  rely  on  a  single  vendor  for  critical  functionality,  always  have 
alternate  products  lined  up 

•  Consider  the  likelihood  that  the  vendor  will  not  be  there  to  support  it  in  the 
future 

Table  9  provides  the  risk  profile  for  the  Anny’s  Common  Software  program  with 
the  following  high  risks  factors  being  identified.  The  complete  results  from  the 
questionnaire  are  contained  in  Appendix  H. 

•  The  implementation  team  has  no  experience  with  the  commercial  product 

•  Many  functions  are  supported  by  the  commercial  product 

•  New  or  immature  commercial  product 

•  Program  has  not  gather  any  information  from  other  organizations  that  have 
implemented  commercial  products 

•  Testing  approach  was  designed  for  traditional  testing  of  a  system 

•  No  contingency  plan  for  commercial  vendor  going  out  of  business 
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•  Other  contractors  supporting  the  organization  in  functions  affected  by  the 
commercial  product  have  no  experience  (new  commercial  product) 


Total  Calculations 

a’s 

b’s 

c’s 

Totals  from  Process: 

9 

8 

1 

Totals  from  Technology: 

4 

6 

2. 

Totals  from  Implementation/ 

f. ; 

1 

A 

Support 

19 

15 

7 

otals 

Program  Totals 

Total  a’s  19  x  (1) 

+  Total  b’s  15  x  (2) 

+  Total  c’s  _7_  x  (3) 

Project  Total  =  _70 


123  82  41 

Table  9.  ARMY  Common  Software  Commercial  Risk  Profile 


H.  SUMMARY 

The  questionnaire  was  implemented  on  current  DOD  projects  that  utilize 
commercial  products.  It  provided  to  be  an  effective  and  efficient  tool  that  could  be 
applied  by  the  program  manager  or  decision  maker  in  a  consistent  and  systematic  manner 
to  assist  them  with  the  early  identification  and  prioritization  of  risks  associated  with 
commercial  products.  This  provides  the  program  manager  or  decision  maker  with 
sufficient  information  and  expands  the  time  they  will  have  to  mitigate  and  resolve  the 
risks  by  letting  them  make  wise  decisions  and  apply  proper  resources,  based  on  the 
prioritization,  early  in  the  process  and  use  their  creativeness  to  properly  mitigate  and 
resolve  each  risk. 
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Identifying  and  understanding  the  risks  of  commercial  products  is  the  first  step  to 
ensure  that  the  acquiring  activity  can  minimize  downstream  surprises  and  problems  and 
achieve  the  benefits  of  using  commercial  products.  Shrinking  budgets  and  tighter 
schedules  virtually  eliminate  any  margins  that  could  be  retained  to  offset  problems  that 
might  occur  later  in  a  program.  The  next  chapter  provides  some  strategies  and  offers 
suggestions  to  help  organizations  manage  and  mitigate  the  risks  associated  with 
commercial  products. 
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VI.  MITIGATION  STRATEGIES  AND  SUGGESTIONS 


A.  INTRODUCTION 

Once  the  risks  have  been  identified,  mitigation  strategies  to  reduce  or  eliminate 
the  risks  need  to  be  developed  and  managed.  Even  though  there  are  many  risks,  there  are 
specific  ways  that  the  risks  can  be  reduced.  This  section  describes  some  mitigation 
strategies  and  offers  some  techniques/suggestions  to  help  organizations  tackle  the 
risk/challenges  listed  in  the  previous  chapter. 

B.  SYSTEM  REQUIREMENTS 

1.  Flexible  and  Negotiable 

A  traditional  development  model  that  specifies  all  system  requirements  prior  to 
considering  the  capabilities  available  in  the  marketplace  is  ill  suited  to  the  development 
of  systems  incorporating  commercial  items.  Since  product  development  is  based  on 
commercial  market  needs  and  is  under  the  manufacturer’s  control,  requirements  must  be 
written  with  a  quantifiable  range  of  acceptable  performance  limits  (e.g.,  high  and  low 
values)  offering  flexibility  and  letting  the  integrator  make  the  best  possible  match  (within 
constraints)  between  commercial  product  capabilities  and  the  requirements. 

The  requirements  should  also  be  prioritized  with  criteria  established  to  distinguish 
absolute  requirements  (must  have)  from  the  less  absolute  (nice  to  have)  requirements; 
providing  the  needed  flexibility  when  selecting  among  a  variety  of  commercial  product 
candidates.  This  flexibility  is  especially  important  with  commercial  products  since  they 
are  sold  and  supported  by  manufacturers  in  an  “as  is”  state  which  may  not  meet  all  the 
requirements  as  stated.  Government  and  commercial  programs  that  were  successful  in 
incorporating  commercial  products  were  able  to  trade-off  requirements  with  the 
operational  and  acquisition  communities  in  order  to  achieve  a  best  value  solution.  For 
example,  The  AWACS  program  eliminated  salt  spray  requirement  to  facilitate  the  use  of 
commercial  computers,  while  the  NSSN  submarine  structure  was  modified  to  provide 
environmental  isolation  [USAFOO].  Adapt  the  requirements  to  meet  the  commercial 
product  is  better  than  modifying  the  commercial  product  to  the  requirements  to  meet  the 
requirements..  If  a  commercial  product  is  modified  to  meet  a  requirement,  the  warranty 
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would  be  voided  and  a  unique  product  may  be  created  that  requires  uniquely  developed 
(and  often  more  costly)  life  cycle  support. 

2.  Examine  Requirements  Gap 

Ideally,  the  commercial  product(s)  functionality  should  closely  match  the 
intended  use.  However,  because  no  commercial  product(s)  have  been  specifically 
designed  to  meet  your  organization's  unique  requirements,  there  will  be  a  gap  between 
the  business  processes  supported  by  your  existing  systems  and  those  supported  by  the 
commercial  product(s).  It  is  imperative  that  you  understand  this  gap  well  before  the 
implementation  begins.  If  this  gap  is  too  great  then  more  effort  will  be  expended 
developing  adequate  interfaces  than  developing  the  product  from  scratch. 

3.  Involve  Users 

Because  the  implementation  of  a  commercial  product  significantly  impacts  the 
functions  of  an  organization,  it  is  imperative  to  involve  the  user  community  in  the 
planning  process  from  the  outset.  Early  end-user  involvement  is  a  common  risk 
mitigation  strategy  to  ensure  that  the  requirements  accurately  reflect  user  needs.  User 
familiarization  allows  for  requirement  prioritization  and  the  early  identification  and 
resolution  (trade-offs)  to  minimize  user  acceptance  issues  and  avoid  costly  changes  and 
delays  during  system  development  and  deployment  activities.  It  also  allows  the  user 
community  the  time  to  become  familiar  with  commercial  technologies  that  are  available 
for  meeting  their  needs.  Once  a  system  is  placed  in-service,  fonnal  user  participation  is 
continued  as  the  system  evolves  and  undergoes  changes  and  updates  requiring 
commercial  products.  In  addition  to  the  technical  issues,  understanding  the  business 
issues  will  lower  the  risks  associated  with  the  commercial  implementation.  A  stable 
operating  environment  coupled  with  the  users  willing  to  accept  a  new  way  of  doing 
business  will  also  minimize  implementation  obstacles. 

Suggestions 

The  following  suggestions  can  be  helpful  for  the  requirements  specifications  of 
commercial  items: 
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To  develop  flexible  and  negotiable  requirements: 

•  Requirements  must  be  written  with  a  quantifiable  range  of  acceptable 
performance  limits. 

•  Identified,  prioritized,  and  negotiated  all  absolute  requirements  (must 
have)  and  plan  to  make  repeated  tradeoffs  among  the  requirements  and  the 
capabilities  in  the  marketplace. 

•  Document  all  tradeoffs  made. 

To  bridge  requirements  gap: 

•  Determine  the  gap  between  the  capabilities  and  services  provided  in  the 
marketplace  and  those  required  by  the  system. 

•  Include  the  vendor  in  tradeoff  discussions  when  possible. 

•  Provide  incentives  to  encourage  the  contractor  to  investigate  all  solutions 
that  lead  to  the  appropriate  outcome. 

•  Ask  vendor  to  conduct  demonstrations. 

•  Don’t  modify  the  commercial  item.  Encourage  vendors  to  change  product 
capabilities  and  performance  to  meet  your  requirements  only  if  the  change 
will  be  incorporated  into  the  commercial  product(s). 

To  Involve  Users: 

•  Communicate  and  cooperate  early  with  the  user  to  ensure  flexibility  in  the 
system  requirements  and  to  share  knowledge  of  potential  commercial 
product(s)  that  may  be  available  to  meet  requirements. 

•  Provide  early  functional  demonstrations,  prototyping  to  provide  user 
hands  on  familiarization  with  the  capabilities  of  the  candidate  commercial 
product(s)  and  get  user  buy-in  on  contract  requirements. 


C.  EVALUATION  OF  COMMERCIAL  PRODUCT  (S) 

It  is  critically  important  to  evaluate  all  aspects  of  a  commercial  item.  In  some 
cases,  commercial  item  evaluations  are  performed  as  part  of  source  selection.  This  is  a 
highly  constrained  form  of  evaluation  that  must  be  conducted  only  in  accordance  with 
source  selection  criteria  and  the  source  selection  plan.  However,  the  definition  of 
evaluation  applied  in  this  document  is  far  broader.  Evaluation  is  also  necessary  to  assist 
in  identifying  commercial  capabilities  such  as  security  and  information  assurance,  inter¬ 
operability,  reliability,  and  maintainability  when  choosing  among  alternate  architectures 
and  designs,  in  determining  whether  new  releases  continue  to  meet  requirements,  and  in 


61 


ensuring  that  the  commercial  items  function  as  expected  when  linked  to  other  system 
components.  These  forms  of  commercial-item  evaluation  provide  critical  information 
about  the  tradeoffs  among  system  context,  architecture  and  design,  and  commercial 
capabilities.  Unfortunately,  evaluating  commercial  items  in  order  to  identify  system 
tradeoffs  is  an  unfamiliar  process  for  many  program  managers  (and  their  users).  It  is 
equally  unfamiliar  for  many  contractors  who  are  more  comfortable  with  simply  meeting  a 
specified  set  of  requirements. 

1.  Market  Research 

Successful  systems  that  incorporate  commercial  items  recognize  that,  as 
customers,  they  must  be  informed  and  assertive  to  maximize  the  benefits  of  using 
commercial  items  [ALBE03]  by  attempting  to  gain  leverage  on  the  vendors  through 
market  research.  Market  research  is  a  process  of  collecting  infonnation  about  existing 
and  emerging  technologies,  products,  manufacturers,  suppliers  and  their  trends.  It 
consists  of  market  surveillance  and  market  investigation.  Market  surveillance  is  a 
continuous  canvassing  of  the  commercial  market  to  identify  existing  and  future 
technologies,  vendors’  products  and  market  trends.  It  can  include  Internet  searches, 
attending  trade  shows,  reading  technology  publications,  hiring  consultants,  visiting 
manufacturer/supplier  facilities  and  product  demonstrations.  Market  investigation  is  a 
more  focused  process  of  identifying  and  detennining  if  specific  commercial  items  can 
meet  particular  functional  requirements.  It  involves  not  only  identifying  potential 
alternatives  for  investigation,  but  also  identifying  any  deficiencies  in  the  commercial  item 
that  would  require  modification,  and  then  determining  the  extent  of  that  modification. 
Extensive  modification  of  commercial  items  within  a  program  would  take  the  product  out 
of  the  category  of  a  commercial  item,  thus  increasing  the  program’s  risk. 

Market  Investigations  also  includes  proactively  planning  for  the  continued 
support  or  replacement  of  soon-to-be  obsolete  products,  identifying  the  product(s)  end  of 
life  (EOL)  and  end  of  service  (EOS)  dates.  One  program  selected  commercial  items  with 
the  expectation  that  the  vendor  would  provide  the  necessary  maintenance  capabilities. 
However,  the  vendor’s  commercial  support  strategy  did  not  provide  the  spares,  training, 
or  repair  cycles  necessary  for  military  use.  The  program  was  left  with  a  choice:  redesign 

the  system  or  buy  the  additional  capability.  [USAFOO] 
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2.  License  and  Data  Rights 

Licensing  is  the  primary  vehicle  for  securing  the  use  of  commercial  items  such  as 
software;  data  rights  are  marketplace  vehicles  for  protecting  a  vendor’s  intellectual 
property.  Understand  completely,  the  details  associated  with  the  product  contract, 
including  the  licensing  agreement.  Vendors  offer  different  licensing  agreements  and  you 
need  to  select  the  agreement  appropriate  to  your  program  and  circumstances.  For 
example,  if  everyone  in  the  organization  will  need  to  access  the  product,  ensure  the 
license  is  for  the  entire  enterprise.  Be  sure  to  find  out:  who  owns  the  license  to  the  source 
code;  what  rights  are  provided  relative  to  source  code  modification;  and  what 
arrangements  will  exist  at  contract  expiration.  One  program  expressed  frustration  that  the 
de  facto  selection  of  a  commercial  item  had  already  been  made  prior  to  release  of  the 
solicitation  because  of  the  beneficial  pricing  arrangements  from  previously  negotiated 
enterprise  licenses.  While  the  larger  organization  saved  money  in  negotiating  one  set  of 
licenses  covering  use  by  many  programs,  this  practice  limited  the  individual  program’s 
flexibility  in  choosing  the  most  appropriate  commercial  item  for  the  system.  Another 
program  neglected  to  negotiate  for  all  necessary  licenses  as  part  of  the  initial 
procurement.  After  the  commercial  item  was  selected  and  system  development  began,  the 
vendor’s  price  for  additional  licenses  increased  dramatically  [USAFOO]. 

Some  commercial  products  are  so  critical  to  the  system  that  the  program  must  be 
protected  from  a  vendor’s  potential  unwillingness  or  inability  to  support  older  releases  of 
the  product.  Some  programs  found  that  an  agreement  to  put  technical  data  in  an  escrow 
account  (rather  than  purchasing  technical  data  directly)  was  a  cost-effective  compromise. 
However,  one  program  never  checked  that  the  escrow  account  was  set  up  and  maintained 
by  the  vendor.  When  the  vendor  went  out  of  business,  the  program  was  forced  to  gather 
what  technical  data  it  could  from  personnel  who  had  previously  worked  for  the  vendor 
[DODAOO].  On  the  other  hand,  successful  programs  negotiated  terms  of  the  escrow  to 
include  the  essential  data  and  contingencies,  audited  the  escrow  account  regularly  to 
make  sure  the  data  was  current  and  complete,  and  budgeted  for  the  cost  of  the  escrow 
throughout  the  life  of  the  system. 
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3.  Vendor  Relationship 

It  is  incumbent  on  the  program  manager  to  determine  how  important  the  program 
is  to  a  specific  vendor  as  part  of  the  commercial-product  evaluation.  The  relationship 
between  the  program  manager  and  the  vendor  is,  in  most  instances,  very  different  from 
the  relationship  with  a  contractor.  While  contract  incentives  shape  the  relationship  with  a 
contractor,  the  vendor  is  selling  a  product.  Program  managers  should  obtain  commercial 
product(s)  from  vendors  who  view  the  software  as  one  of  their  products.  If  a  vendor  does 
not  consider  an  item  to  be  part  of  its  business,  and/or  developing  software  products  is  not 
the  main  business  of  the  vendor,  then  its  development,  release  management,  and 
documentation  practices  may  not  be  adequate. 

Program  managers  need  to  develop  a  trusting,  but  contractual  and  mutually 
beneficial  relationship  with  the  vendor.  Finding  vendors  with  the  best  quotes  and  support 
services  are  time  consuming  and  not  an  easy  process.  Assess  the  vendor’s  past 
experience  employing  commercial  products.  When  both  government  and  the  vendor 
were  experienced  in  applying  commercial  products,  the  success  rate  was  high  and  cost 
savings  were  dramatic.  When  both  were  inexperienced  the  success  rate  was  very  low  and 
costly,  many  times  resulting  in  substantial  overruns  and  even  program  failure  [FAAC02]. 

Consider  the  characteristics  of  the  vendor  as  part  of  the  process.  Examples  of 
characteristics  to  look  for  include  company  size,  their  level  of  establishment,  how  long 
they  have  been  selling  the  product,  its  level  of  support  (does  it  continue  to  support 
customers  using  significantly  older  versions  of  their  product  as  well  as  the  customers 
using  the  latest  version),  and  is  the  company  willing  to  work  with  the  organization  rather 
than  for  it  (will  they  maintain  their  autonomy  and  listen  to  the  organization’s  requests  for 
enhancements  rather  than  automatically  accepting  that  such  changes  must  be  made?). 

Figure  out  the  leverage  you  have  and  how  you  can  influence  the  vendor  to  be 
responsive  to  unique  program  needs  and  incorporate  new  features  into  the  commercial 
item.  Some  program  managers  have  expressed  frustration  that  vendors  do  not  react  to 
program  needs  and  direction.  Other  program  managers  have  tried  to  use  the  same 
techniques  with  vendors  that  had  been  successful  when  applied  to  contractors  and 
subcontractors — usually  with  disappointing  results  [DODAOO].  The  degree  of  leverage 
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you  can  apply  to  your  vendors  varies  depending  on  the  size  of  your  program  and  the 
presence  your  organization  has  in  the  industry.  The  larger  your  organization  the  more 
pressure  you  can  exert  in  getting  vendors  to  meet  your  projects  requirements. 

One  approach  to  this  is  to  join  the  vendor  users’  group  and  the  appropriate 
professional  and  standards  bodies.  Joining  the  users'  group  is  valuable  as  it  provides  the 
organization  some  insight  into  the  use  of  the  product  in  other  parts  of  the  marketplace. 
Future  directions  of  the  product  may  be  discussed  in  these  meetings,  providing  the 
organization  with  the  opportunity  to  express  their  needs  in  the  domain  the  product 
addresses.  Joining  the  relevant  professional  and  standards  bodies  is  important  so  that  the 
program  can  remain  current  with  the  direction  being  taken  that  affects  their  system. 
Further,  by  being  active  in  the  professional  and  standards  bodies,  the  organization  is  able 
to  influence  (although  not  control)  the  future  course  of  the  business  approach  so  that  the 
business  practices  embodied  by  the  philosophy  are  more  likely  to  remain  consistent  with 
the  organization's  needs. 

4.  Testing,  Evaluation,  and  Validating  Reliability 

Given  the  lack  of  technical  information  about  commercial  product(s)  (i.e.,  “black 
box”  -  undisclosed  designs)  and  the  variety  of  product  types  that  change  rapidly,  testing 
potential  commercial  product(s)  is  a  necessary  step  in  product  evaluation  to  ensure  that 
operational  suitability;  reliability,  availability  and  maintainability  requirements  are  met, 
since  manufacturer  claims  about  the  capabilities  tend  to  be  optimistic.  More  effective  up¬ 
front  planning  of  independent  test  and  evaluation  is  needed  to  ensure  that  enough  data  is 
obtained  to  fully  evaluate  the  capabilities.  Exercise  a  healthy  skepticism  of  commercial 
product  claims  by  testing  candidate  products  using  an  application  testbed  to  verify 
features  and  to  support  your  selection  decision. 

Full  system-level  testing  must  be  performed  in  a  test  facility  that  provides  or 
emulates  the  external  interfaces  and  actual  operating  environment  in  order  to  verify 
operational  suitability,  effectiveness  and  performance  and  should  never  be  waived.  This 
raises  the  probability  that  the  commercial  product(s)  perform  as  they  did  in  the 
development  environment  and  that  they  do  not  introduce  any  unknown  perfonnance 
characteristics  into  the  interfacing  systems.  As  engineering  changes  are  introduced  into 
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the  continuously  evolving  system,  a  dedicated  testing  environment  must  be  maintained  to 
replicate  the  integration  steps  and  determine  if  the  new  product  or  integrated  change  has 
affected  the  performances  of  the  system. 

Confirm,  with  other  users,  the  product's  capabilities,  especially  performance, 
reliability,  and  scalability  by  ensuring  that  the  product's  capabilities  support  the  needs  of 
your  organization.  For  instance,  confirm  that  the  product  has  previously  supported  the 
number  of  users  and  geographic  locations  your  organization  will  require. 

Reliability  requirements  must  be  established  early  in  order  to  insure  adequate 
testing  and  verification  of  the  reliability  of  commercial  items.  Since  a  commercial  item 
has  already  been  designed  and  developed,  and  its  reliability  already  established  by  the 
vendor,  the  reliability  verification  should  be  an  assessment  of  the  product  within  the 
military  wartime  environment  in  which  it  will  be  used.  Testing  the  commercial  item  in 
your  operating  environment  to  verify  that  the  item’s  reliability  is  meeting  the  user’s 
requirements.  Lower  reliability  greatly  impacts  the  support  costs,  system  availability, 
and  thus  the  mission  accomplishment. 

Suggestions 

The  following  suggestions  can  be  helpful  for  evaluating  commercial  items: 

•  Employ  outside  experts  to  support  program-office  evaluation  activities. 

•  Train  the  program  office  and  the  stakeholders/users  on  how  to  evaluate 
commercial  items. 

•  If  possible,  obtain  hands-on  experience  with  the  system.  Consider 
prototyping  or  piloting  the  commercial  item  in  your  environment.  Ask 
vendor  for  a  demonstration. 

•  At  a  minimum,  visit  another  organization  that  is  operating  the  same 
software. 

•  Decide  in  advance  what  infonnation  you  want  to  gain  from  the  evaluation 
of  a  commercial  item. 

•  Unless  it  is  impractical,  evaluate  potential  commercial  items  in  a  system 
test  bed.  Do  not  consider  buying  any  commercial  product  you  haven’t 
demonstrated  in  house. 
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To  understand  the  marketplace: 

•  Conduct  market  research  independent  of  the  contractor  to  capture  current 
information  and  base  market  research  decisions  on  fairness,  competition, 
and  ethics. 

•  Participate  in  the  relevant  conferences,  trade  shows,  and  user, 

professional,  and  standards  groups. 

•  Allocate  resources  for  marketplace  activities. 

•  Proactively  anticipate  obsolescence  situations  due  to  rapid  and 

asynchronous  product  changes. 

•  Assess  the  total  system  operation  and  support  effectiveness,  not  just 
system  perfonnance  by  determining  the  projected  manufacturer  support 
period  and  inventories  for  a  particular  product. 

•  Select  a  reputable  and  reliable  vendor  that  plans  to  be  available  to  support 
the  product. 

To  help  with  Licenses  and  Data  Rights: 

•  Thoroughly  understand  commercial  products  licensing  terms  and 

conditions  to  ensure  warranties  are  enforceable. 

•  Contracting  officers  should  consider  including  contract  options(s)  for  parts 
and  technical  buyouts  to  support  future  logistics  requirements. 

•  Licenses  that  transfer  to  the  government/maintainers. 

•  Volume  discounts. 

•  Put  technical  data  into  escrow  accounts.  Regularly  audited  the  accounts 
to  make  sure  the  data  I  complete  and  current  and  complete. 

To  strengthen  vendor  relationships: 

•  Verify  the  claims  made  for  commercial  items  by  vendors  and  contractors. 

•  Verify  the  availability  of  commercial  items. 

•  Examine  any  acquisition  strategy  to  see  where  it  can  be  made  more 
flexible  or  better  suited  to  the  unique  commercial  aspects  of  the  system  in 
question. 

•  Use  contract  incentives  to  encourage  appropriate  relationships  . 

•  Maintain  close  relationships  with  vendors  to  exploit  improvements  and 
avoid  surprises. 
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To  test  commercial  items  and  validate  reliability: 

•  Do  not  rely  on  vendor  claims,  verify  with  operational  demonstrations  as 
early  as  possible  by  evaluating  the  potential  commercial  products  in  a 
system  test  bed  to  ensure  product(s)  are  compliance  with  performance 
requirements. 

•  Focus  test  beds  on  high-risk  items. 

•  Test  for  unanticipated  side  effects  in  areas  such  as  security,  safety, 
reliability  and  perfonnance  from  commercial-item  upgrades.  Ensuring  all 
system  configurations  (more  are  possible  with  commercial  product  use) 
can  be  recreated;  and  determine  if  new  manufacturer  changes  to  a 
commercial  product  configuration  cause  any  unforeseen  impacts  (i.e., 
“unknown-unknowns”  to  system  perfonnance). 

•  Ensure  that  performance  pass  or  fail  criteria  are  clearly  specified  in  the 
contract. 

•  Contracting  offices  should  require  contractors  to  fully  disclose  item 
reliability  data. 

•  Test  Organizations  should  validate  the  reported  reliability  of  commercial 
item  components  and  test  them  thoroughly  in  new  operational 
environments. 

•  Select  a  mature  product.  Implementing  a  commercial  item  with  a 
successful  track  record  is  less  risky  than  one  that  involves  new,  unproven 
capabilities. 


D.  TECHNOLOGY 

The  technology  as  well  as  your  ability  to  deal  with  it  may  both  be  immature  and 
change  rapidly.  Planning  ahead  for  technology  insertion  should  be  an  integral  part  of 
your  program.  One  has  to  predict  the  change  cycle  for  each  imbedded  commercial 
product  and  plan  for  regular  refreshment  of  the  system  throughout  the  design, 
development,  production  and  sustainment  phases  of  the  program. 

1.  Integration 

Integrating  commercial  product(s)  into  a  functional  system  presents  new 
challenges  and  projects  cannot  be  treated  as  nonnal  low-risk  commercial  item 
acquisitions.  Different  vendors  write  different  components  with  different  needs  in  mind 
that  may  need  to  be  adapted  to  work  properly.  Integration  efforts  may  not  only  require 
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research  and  development,  but  also  may  need  both  a  demonstration  and  a  full-scale 
development  phase  before  production 

Although  the  expertise  is  growing,  relatively  few  programs  or  contractors  have 
extensive  experience  integrating  commercial  items  into  DOD  systems.  Knowledge  of 
both  the  system  context  and  each  selected  commercial  item  is  necessary.  One  program 
assumed  that  heterogeneous  commercial  items  could  be  integrated  with  relatively 
minimal  effort.  The  program  neglected  the  hard  engineering  work  needed  to  develop 
realistic  integration  and  test  schedules,  to  specify  acceptance  criteria  for  the  system,  or  to 
plan  for  long-term  system  evolution  [DODAOO].  These  oversights  resulted  in  unhappy 
users,  finger-pointing  between  the  vendors  and  the  program  office,  and  cost  and  schedule 
overruns.  Several  other  programs  found  that  unique  technical  expertise  was  required  to 
integrate  commercial  items  because  the  internal  architectural  and  usage  assumptions  of 
the  items  were  unknown. 

Three  integration  techniques  for  commercial  items  are: 

•  To  wrap  the  items  in  a  software  container. 

•  Use  glueware  to  mediate  item  interactions. 

•  Using  bridges  or  adaptors  to  smooth  over  incompatibilities  in  the  item 
interfaces. 

All  of  these  are  “black-box”  techniques  that  can  be  applied  without  access  to  the 
commercial  product(s)  source  code.  Wrappers  are  software  containers  used  to  mediate 
access  to  the  commercial  items.  They  can  be  used  to  force  compliance  to  programming 
standards,  provide  a  standard  interface  to  the  commercial  product(s),  restrict  the  available 
functionality  of  the  commercial  item,  or  can  be  upgraded  or  swapped  out  with  a  different 
vendors  commercial  item  [MAUROO].  Glueware  is  used  as  middleware  to  bind 
components  together.  It  can  be  used  for  control  flow,  to  invoke  the  item’s  functionality 
and  do  exception  handling.  This  can  also  include  code  to  resolve  incompatibilities 
among  commercial  items  [MAUROO],  By  acting  as  an  adaptor  or  bridge,  the  glueware 
can  allow  the  interaction  of  two  items  [MAUROO], 
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2.  Obsolescence  and  Upgrades 

Many  organizations  or  program  office  assume  they  can  avoid  the  problems 
associated  with  upgrade  by  simply  continuing  to  deploy  an  older  release  of  a  commercial 
item.  While  this  may  be  true  for  some  hardware  items,  it  is  rarely  the  case  for  software 
items,  where  new  and  desirable  capabilities  and  perfonnance  are  frequently  added,  bugs 
are  fixed,  and  vendors  drop  maintenance  for  older  releases.  A  release  or  two  can 
sometimes  be  safely  skipped,  but  most  software  commercial  items  (and  many  hardware 
commercial  items)  must  eventually  be  upgraded.  Except  in  very  specific  cases,  DOD  is 
normally  ill  prepared  to  implement  the  necessary  changes  to  old  versions  of  commercial 
items  in  order  to  avoid  technical  obsolescence  and  keep  them  functioning,  even  when 
good  technical  data  is  available.  Several  programs  were  successful  by  deliberately  pre¬ 
planning  for  frequent  upgrades  of  commercial  items,  technology  insertion,  and  retirement 
of  obsolete  items  [DODAOO].  Of  course,  even  the  most  careful  planning  cannot 
anticipate  all  exigencies,  such  as  a  vendor  going  out  of  business  or  being  taken  over  by  a 
larger  firm  with  different  priorities 

3.  New  and  Strong  Configuration  Management  Techniques 

The  rapid  evolution  and  proprietary  nature  of  commercial  products/systems 
require  a  robust  and  diligent  configuration  management  (CM)  program.  Frequent 
changes  to  commercial  items  have  caused  many  programs  to  maintain  multiple 
configuration  baselines  both  during  development  and  in  the  field.  This  places  unusual 
demands  on  traditional  configuration-management  processes  that  strive  to  maintain  a 
single  configuration  baseline.  Several  programs  that  depended  on  multiple  commercial 
items  found  that  some  items  required  specific  versions  of  other  items  in  order  to  interface 
effectively.  Upgrading  one  commercial  item  caused  a  chain  reaction  that  demanded 
changes  to  other  commercial  items  within  the  system  [DODAOO]. 

Unlike  custom  developed  systems,  the  government  has  no  control  over  the  speed 
and  content  of  product  configuration  changes  since  the  commercial  product 
manufacturers  control  them.  Vendors  release  items  according  to  their  own  schedules,  and 
many  programs  found  that  individual  sites  were  not  always  willing  to  upgrade  to  the 
latest  version.  This  requires  programs  to  possess  a  configuration-management  system 
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that  lets  then  select  from  among  multiple  versions  of  commercial  items  in  order  to 
construct  different  system  configurations. 

Commercial  products  are  proprietary  to  the  manufacturer  and  get  documented  at  a 
higher  level  (source  control  drawings,  specification  sheets,  inputs/outputs,  etc.)  resulting 
in  limited  information  on  manufacturing  processes,  internal  design,  components,  etc. 
Included  with  this  higher  level  of  documentation  are  different  numbering  conventions  by 
the  manufacturers.  These  differences  shift  CM  focus  from  controlling  configurations  (as 
with  custom  development  programs)  to  managing  commercial  product  and  system 
configurations  (at  the  manufacturer-controlled  product  level). 

Suggestions 

The  following  suggestions  can  help  minimize  the  impact  of  commercial 
technology: 

To  integrate  product(s) 

•  Establishing  an  ongoing  market  research  effort  that  includes  market 
surveillance  (technologies,  trends,  available  manufacturers  and  products, 
etc.)  and  market  investigation  (product  testing  and  obsolescence 
projections). 

•  Developing/delivering  integrated  technology  evolution  planning  data, 
conducting  working  group  meetings  and  providing  status  at  program 
reviews. 

•  Base  interfaces  on  publicly  recognized  industrial  standards  that  are  widely 
supported  in  the  marketplace. 

•  No  vested  interest  in  any  one  particular  manufacturer  or  commercial 
product  set. 

To  manage  change: 

•  Ensure  that  rigorous  configuration  management  is  exercised. 

•  Monitor  the  marketplace  for  technology  advancements. 

•  Establish  plans  to  work  with  vendors  for  problem  resolution. 

•  Periodically  check  with  vendors  for  planned  software  upgrades/updates  to 
commercial  products. 

•  Developing/delivering  periodic  (every  four  months)  system  product 
obsolescence  projections,  impacts  and  solutions. 
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Establishing  a  test  facility  capable  of  continuously  testing  commercial 
products  for  compatibility,  compliance  and  conformance. 


E.  AVOIDING  MODIFICATION 

Modification  of  commercial  products  can  involve  the  addition  or  deletion  of  code, 
changes  to  the  hardware  design,  or  changes  to  any  of  the  product  support  (documentation, 
spares,  etc.).  Commercial  products  must  be  used  “unmodified”  to  retain  the  value  of  a 
commercial  product;  otherwise,  voided  warranties,  lack  of  support,  and  upgrade 
difficulties  will  result.  If  this  strategy  is  ignored  the  program  runs  the  risk  of  incurring 
additional  support  costs  and  supportability  issues  that  may  be  less  cost  effective  the 
custom  design.  The  savings  in  development  costs  and  schedules  would  be  offset  by  the 
modifications  of  the  commercial  products  resulting  in  a  unique  version  of  the  product  that 
the  manufacturer  will  not  support  under  warranty  and  must  be  supported  separately  from 
other  versions,  often  with  increased  support  costs.  One  private  corporation  fell  into  the 
trap  of  modifying  most  of  its  commercial  items  in  order  to  give  them  a  unique  corporate 
flavor.  As  a  result  of  the  practice,  many  of  the  corporate  programs  modifying 
commercial  items  experienced  recurring  technical  problems  and  cost  overruns 
[DODAOO], 

Suggestions 

The  following  suggestions  can  help  organizations  avoid  modifications  to 
commercial  product(s): 

•  Persuading  the  manufacturer  to  incorporate  the  acquiring  activities  unique 
requirements  as  part  of  the  commercial  product’s  functionality; 

•  Verify  that  the  contractor  proposes  to  use  the  commercial  product  without 
modification 

•  Requiring  that  notification  of  proposed  COTS  modifications  be  made  only 
with  trade-off  considered  and  Government  consent; 

G.  TOTAL  OWNER  COST  (TOC) 

Both  commercial  and  DOD  programs  frequently  underestimate  the  unique 
sustainment  costs  associated  with  commercial  items.  These  costs  include  market 
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research,  evaluation,  test  and  integration  for  version  upgrade,  commercial-item 
replacement,  technology  refresh,  and  annual  licensing  fees.  Programs  must  deal  with 
change  and  expense  elements  throughout  a  product  lifecycle  that  may  not  be  evident  in  a 
custom  product.  Successful  programs  have  generated  strategies  for  assessing  items  such 
as  product  deviations  caused  by  commercial  product  evolution  cycles,  extended  military 
product  support  and  licensing  against  the  expected  benefits  of  more  affordable  sparing, 
shorter  development  times  and  increased  performance  [USAFOO]. 

Suggestions 

The  following  suggestions  can  help  organizations  identify  Total  Ownership  Cost 

(TOC): 

•  Identify  and  budget  sufficient  funds  and  staff  for  monitoring  current  and 
emerging  commercial  product(s)  and  technology  -  market  research, 
integration  lab,  testing  facility,  license  renewal  and  data  rights,  and 
reacting  to  new  product  releases  -version  upgrades  in  annual  budgets 

•  Incentive  the  prime  contractor  to  provide  a  credible  estimate  of  support 
costs 

•  Use  multi-year,  unrestricted  contracting  could  potentially  reduce  costs 
even  more. 

H.  SUMMARY 

Unfortunately,  there  are  no  silver  bullets  to  resolve  the  risks/challenges  associated 
with  commercial  items.  Early  identification  of  the  risks  associated  with  commercial 
items  and  the  techniques  and  suggestions  discussed  in  this  chapter  provide  an  effective 
approach  that  can  be  incorporated  into  an  overall  risk  management  program  for  systems 
employing  commercial  items.  Such  a  risk  mitigation  approach  allows  the  acquiring 
activity  to  benefit  from  commonly  experienced  government  and  industry  lessons-learned. 
Personnel  will  become  better  educated,  trained,  and  effectively  employ  commercial  items 
by  actively  soliciting  and  rigorously  incorporate  into  their  plans  those  lessons  learned 
from  organizations  similar  to  theirs  and  by  exploring  products  before  selecting  them, 
talking  to  some  of  the  product's  other  customers  and  understanding  the  product's 
customer  base.  Allowing  them  to  effectively  and  efficiently  employ  commercial  items 
within  their  programs. 
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VII.  CONCLUSION 


The  inclusion  of  commercial  items  in  the  acquisition  process  is  recognized  as  an 
opportunity  to  save  both  time  and  money.  But  it  is  not  the  Holy  Grail.  Anyone  who 
believes  that  selection  of  a  commercial  product  and  inserting  it  into  a  program  will  be  the 
quick  fix  believes  in  fairy  tales,  and  does  not  really  understand  the  process.  Commercial 
products  can  do  all  that  everyone  expects  them  to  do,  and  may  be  an  excellent  solution  in 
many  cases,  but  their  use  should  be  the  result  of  careful  analysis  and  research.  There  is  a 
tendency  to  assume  that  a  commercial  product  can  be  used  as-is,  without  any  serious 
thought  given  to  the  difficulty  and  risk  involved  in  the  commercial  product.  It  has  been 
assumed  that  the  use  of  a  commercial  product  alleviates  all  risk  of  integration,  when,  in 
fact,  just  the  opposite  may  occur:  commercial  products  may  be  even  more  difficult  to 
integrate. 

Every  aspect  of  acquisition  planning,  system  engineering  processes,  test  planning, 
etc.  must  be  explicitly  crafted  to  account  for  every  challenge  the  commercial  product 
presents.  The  mentality  ought  to  be  how  we  can  do  it  as  opposed  to  why  we  cannot. 
Everything  about  the  commercial  product  must  be  known  and  understood  by  those  who 
establish  the  requirements.  Not  every  new  requirement  can  realistically  be  addressed 
with  a  commercial  product.  The  commercial  products  must  be  chosen  carefully  with  the 
marketplace  clearly  understood  in  order  to  have  a  flexible  range  of  “requirements” 
sufficient  to  allow  commercial  products  to  qualify. 

The  risks  associated  with  traditional  system  development  do  not  disappear  simply 
because  the  system  makes  use  of  commercial  products.  Risks  associated  with 
commercial  products  are  likely  to  change  more  rapidly,  and  new  risks  may  arise  more 
often  than  with  customary  system  risks.  In  order  for  systems  to  be  successful  and  achieve 
the  benefits  of  using  commercial  items,  they  need  to  minimize  the  uncertainties  and 
manage  the  unique  risks  associated  with  the  commercial  product(s).  Identification,  the 
first  step  in  the  process,  involves  transfonning  the  uncertainties  and  issues  about  the 
project  into  distinct  (tangible)  risks  that  can  be  described  and  measured.  With  any  risk, 
awareness  of  lessons  learned  by  other  organization  that  have  implemented  systems  using 
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commercial  products  will  help  build  or  strengthen  strategies  to  address  any  unexpected 
future  challenges. 

This  thesis  describes  a  framework,  checklist  (questionnaire)  and  provides  some 
strategies  and  suggestions  to  help  organizations  manage  and  mitigate  the  risks  associated 
with  commercial  products.  The  checklist  serves  as  a  starting  point  and  enhances  the 
project  manager’s  or  decision-maker’s  risk  management  abilities  by  providing  a 
systematic  and  repeatable  method,  early  in  the  process,  for  the  identification  of  risks 
associated  with  the  use  of  commercial  products.  It  uses  a  set  of  known  risks  that  are 
classified  into  three  categories:  process,  technology,  and  implementation  and  support. 
Many  organizations  which  attempt  to  implement  a  commercial  product(s)  without 
sufficient  analysis  and  preparation  encounter  significant  challenges  that  can  be  related  to 
the  business  processes  used  to  build  the  systems,  technologies  used  to  construct  the 
systems,  and  logistical  support  and  implementation  issues  that  inevitably  arise.  Each 
category  is  then  further  divided  into  specific  elements  or  attributes  to  generate  the 
projects  risk  profile.  This  profile  detennines  what  level  of  impact  (high,  medium,  or  low) 
these  factors  have  on  the  programs  that  incorporate  the  commercial  product.  Managers 
can  then  prioritize  these  risks  and  apply  resources  to  properly  mitigate  and  resolve  the 
identified  risks. 

To  test  the  checklist,  it  was  implemented  on  active  DOD  programs  that 
incorporate  commercial  products  into  their  systems  and  proved  to  be  an  effective  and 
efficient  tool.  Project  managers  or  decision-maker’s  were  provided  with  sufficient 
information  to  identify  and  prioritize  risks  early  in  the  process.  They  could  then  use  their 
creativity  along  with  some  of  the  suggestions  to  make  wise  decisions  and  positively 
impact  the  success  of  their  programs.  [FALV02,  FLAN02,  FIAKE03,  PORT03,  TRIE02, 
WILL03] 

While  considerable  work  still  remains  to  be  done  in  developing  additional 
identification  methods,  analysis,  planning,  tracking,  and  controlling  risks  associated  with 
commercial  products.  The  project  manager  or  any  decision  maker,  should,  as  a 
minimum,  fill  out  and  discuss  the  checklist,  with  the  checklist  becoming  a  natural  part  of 
the  project  activities.  This  will  force  program  managers  to  focus  their  efforts  on  the 
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highest  impact  areas  and  eliminate  fires  before  they  happen,  thus,  minimizing  their 
organization's  risk  severity  rating  while  curbing  future  expenditures.  Although  the  scope 
of  the  thesis  ends  at  this  point,  it  is  recommended  that  the  program  or  project  manager’s 
develop  a  risk  plan  for  each  prioritized  risks.  These  identified  risks  would  then  be 
monitored  and  tracked  continuously  throughout  the  life  cycle  of  each  project.  Only  by 
understanding  and  addressing  these  unique  factors  imposed  by  commercial  products  will 
the  program  managers  be  able  to  attain  their  benefits.  Enhancing  their  ability  to  manage 
the  risks  associated  with  commercial  products  and  making  them  more  successful  in  all 
the  software  development  projects  they  lead.  Moving  towards  market-oriented  business 
practices  that  are  better  suited  to  the  acquisition  and  life  cycle  support  of  commercial 
(COTS)-based  systems. 
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APPENDIX  A  FAR  DEFINITION  OF  COMMERCIAF  ITEM 


(a)  Any  item,  other  than  real  property,  that  is  of  a  type  customarily  used  for 
nongovernmental  purposes  and  that  — 

(1)  Has  been  sold,  leased,  or  licensed  to  the  general  public;  or, 

(2)  Has  been  offered  for  sale,  lease,  or  license  to  the  general  public; 

(b)  Any  item  that  evolved  from  an  item  described  in  paragraph  (a)  of  this 
definition  through  advances  in  technology  or  performance  and  that  is  not  yet  available  in 
the  commercial  marketplace,  but  will  be  available  in  the  commercial  marketplace  in  time 
to  satisfy  the  delivery  requirements  under  a  Government  solicitation; 

(c)  Any  item  that  would  satisfy  a  criterion  expressed  in  paragraphs  (a)  or  (b) 
of  this  definition,  but  for 

(1)  Modifications  of  a  type  customarily  available  in  the  commercial 
marketplace;  or 

(2)  Minor  modifications  of  a  type  not  customarily  available  in  the 
commercial  marketplace  made  to  meet  Federal  Government  requirements.  Minor 
modifications  means  modifications  that  do  not  significantly  alter  the  nongovernmental 
function  or  essential  physical  characteristics  of  an  item  or  component,  or  change  the 
purpose  of  a  process.  Factors  to  be  considered  in  detennining  whether  a  modification  is 
minor  include  the  value  and  size  of  the  modification  and  the  comparative  value  and  size 
of  the  final  product.  Dollar  values  and  percentages  may  be  used  as  guideposts,  but  are 
not  conclusive  evidence  that  a  modification  is  minor; 

(d)  Any  combination  of  items  meeting  the  requirements  of  paragraphs  (a),  (b), 
(c),  or  (e)  of  this  definition  that  are  of  a  type  customarily  combined  and  sold  in 
combination  to  the  general  public; 

(e)  Installation  services,  maintenance  services,  repair  services,  training 
services,  and  other  services  if  such  services  are  procured  for  support  of  an  item  referred 
to  in  paragraphs  (a),  (b),  (c),  or  (d)  of  this  definition,  and  if  the  source  of  such  services  — 
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(1)  Offers  such  services  to  the  general  public  and  the  Federal 
Government  contemporaneously  and  under  similar  terms  and  conditions;  and 

(2)  Offers  to  use  the  same  work  force  for  providing  the  Federal 
Government  with  such  services  as  the  source  uses  for  providing  such  services  to  the 
general  public; 

(f)  Services  of  a  type  offered  and  sold  competitively  in  substantial  quantities 
in  the  commercial  marketplace  based  on  established  catalog  or  market  prices  for  specific 
tasks  performed  under  standard  commercial  terms  and  conditions.  This  does  not  include 
services  that  are  sold  based  on  hourly  rates  without  an  established  catalog  or  market  price 
for  a  specific  service  performed; 

(g)  Any  item,  combination  of  items,  or  service  referred  to  in  paragraphs  (a) 
through  (f),  notwithstanding  the  fact  that  the  item,  combination  of  items,  or  service  is 
transferred  between  or  among  separate  divisions,  subsidiaries,  or  affiliates  of  a 
contractor;  or 

(h)  A  nondevelopmental  item,  if  the  procuring  agency  determines  the  item 
was  developed  exclusively  at  private  expense  and  sold  in  substantial  quantities,  on  a 
competitive  basis,  to  multiple  State  and  local  governments. 
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APPENDIX  B  COMMERCIAL  ITEM  RISK  QUESTIONNAIRE 


The  purpose  of  this  questionnaire,  which  takes  about  10  minutes  to  complete,  is  to 
identify  and  investigate  the  unique  risks  associated  with  implementing  Commercial-Off- 
The-Shelf  (COTS)  software  application  package(s).  Answering  the  questions  will  help 
you  better  understand  the  overall  level  of  risk  within  your  program.  It  is  recommended 
that  someone  responsible  for  specifying,  procuring  and  developing  software  systems 
should  complete  this  questionnaire.  After  completion,  please  e-mail  the  questionnaire  to 
rcummins@nps.navy.mil.  We  believe  that  you  will  find  the  questionnaire  both 
interesting  and  provocative  and  look  forward  to  receiving  your  reply.  We  appreciate  your 
help  in  our  research  effort,  therefore  if  you  would  like  a  copy  of  our  completed  study 
please  indicate  this  on  the  last  page  of  the  questionnaire. 

Thank  you  in  advance  for  your  time  and  co-operation. 

The  questionnaire  is  divided  into  two  parts: 

Section  I.  Demographic  Information.  Collecting  background  information  about 
the  survey  respondents. 

Sections  II.  Risk  questionnaire,  a  modification  to  the  Information  Technology 
Resources  Board’s  (ITRB)  “Risk  Profile”,  that  is  organized  around  three  broad 
categories:  process,  technology,  and  implementation/logistics  support.  Each  category, 
which  represents  critical  aspects  required  for  the  successful  implementation  of  a 
commercial  application  package(s),  is  defined  below: 


•  Process:  The  key  considerations  for  developing  and  executing  a 
successful  acquisition  process  with  the  system/program  requirements 
driving  the  organization  to  consider  a  commercial  solution  and  the  “fit”  of 
those  requirements  with  available  commercial  application  package(s). 
Key  areas  are  organizational,  planning,  tracking,  contractual  parameters, 
and  evaluation  of  vendor’s  experience  and  past  performance. 

•  Technology:  The  technical  “fit”  of  the  commercial  product(s)  with  the 
existing  and  planned  technical  architecture,  which  supports  an 
organization.  This  includes  the  organization’s  inherent  technical 
challenges,  such  as  the  number  and  complexity  of  interfaces. 
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•  Implementation/Logistics  Support:  The  process  contains  intermediate 
and  final  work  product  characteristics  for  the  delivery  of  a  commercial 
solution  within  an  organization  that  includes  -  but  is  not  limited  to 
performance  measures,  vendors  availability  of  support,  testing  and 
managing  organizational  change. 


SECTION  I 

Service: _ 

Agency  or  Organization: _ 

Project/System  Name: _ 

Telephone: _  Fax: _ 

1 .  Which  category  best  describes  your  main  job  function  in  your  organization? 

a.  Management 

b.  System  analysis  or  design 

c.  Application  or  system  programming 

2.  Have  you  participated  in  selecting  COTS  software  components  that  where  later 
adapted  or  integrated  into  your  project/system?  How  many  times? 


3.  How  long  is  your  work  experience  with  building  system  from  COTS 

components? _ Years 

4.  Please  state  any  good  practices,  or  lessons  you  have  learnt  from  past  experience 
when  acquiring  and  developing  systems  using  COTS  software. 


SECTION  II 

Questions  are  organized  around  the  three  broad  areas  of  implementing  a  COTS 
solution  as  presented  above.  Each  question  prompts  you,  the  respondent,  to  think  about 
key  factors  for  a  successful  COTS  application  package  implementation.  You  should 
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carefully  consider  your  answer  in  terms  of  how  it  pertains  to  projects  within  your  own 
organization. 

Answers  to  each  question  are  provided  by  the  choice  a,  b  or  c,  which  correlate  to 
the  three  levels  of  risk:  low,  medium  and  high,  respectively. 

Process 

1 .  How  well  are  the  requirements  for  your  system/program  documented? 

a.  Thoroughly — comprehensive,  current  documentation  exists 

b.  Moderately  well — comprehensive  documentation  exists,  but  has  not  been 
updated  recently 

c.  Poorly — minimal  documentation  exists 

2.  Because  specific  requirements  are  associated  with  each  COTS  application 
package(s),  how  would  you  describe  the  relationship  between  the  specifications  of  the 
COTS  product(s)  and  the  requirements  of  your  system/program? 

a.  Ideal — great  fit,  fully  meets  requirements 

b.  Satisfactory — acceptable  fit,  meets  most  requirements 

c.  Unsatisfactory — marginal  fit,  must  be  modified  to  meet  requirements 

3.  How  many  COTS  product(s)  can  accommodate  your  system/program 
requirements? 

a.  Many 

b.  Some 

c.  Few 

4.  How  would  you  describe  the  process  by  which  your  organization  will  implement 
new  requirements  after  the  initial  implementation  of  the  COTS  product(s)? 

a.  Well-defined,  proven  process  has  been  established  to  evaluate  and 
implement  new  requirements  (e.g.,  configuration  control  board) 

b.  Process  for  evaluating  and  implementing  new  requirements  has  been 
discussed,  but  not  solidified 

c.  No  process  exists  for  evaluating  and  implementing  new  requirements 

5.  How  would  you  describe  your  system/program’s  ability  to  adapt  to  the  new 
requirements  supported  by  the  COTS  product(s)? 

a.  Very  able — there  is  a  general  understanding  that  the  new  requirements 
would  enhance  organization's  operation 

b.  Somewhat  able — there  is  a  general  understanding  that  the  new 
requirements  would  not  enhance  or  deter  organization's  operation 
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c.  Not  able — there  is  a  general  understanding  that  the  new  requirements 
would  deter  organization's  operation 

6.  How  was  the  COTS  product  evaluated  and  selected? 

a.  Well-defined,  proven  process  has  been  established  to  evaluate  and  select 
COTS  product 

b.  Process  for  evaluating  and  selecting  COTS  products  have  been  discussed, 
but  not  solidified 

c.  Ad  hoc,  no  process  exists  for  evaluating  and  implementing  new 
requirements 

7.  What  is  the  vendor's  experience  with  implementing  the  COTS  product(s)  in 
organizations  of  a  size  similar  to  yours? 

a.  Extensive  experience,  established  company  with  quality  workforce  and 
facilities 

b.  Some  experience 

c.  No  experience,  company  is  start-up  and  situation  is  highly  dynamic 

8.  How  has  the  vendor  performed  in  the  integration  of  the  COTS  application 
package(s)  elsewhere? 

a.  Excellent  past  perfonnance 

b.  Good  past  performance 

c.  Poor  or  unknown  past  performance 

6.  What  is  the  vendor's  experience  with  implementing  the  considered  COTS 
product(s)  in  organizations  of  a  management  structure  similar  to  yours? 

a.  Extensive  experience,  established  company  with  quality  workforce  and 
facilities 

b.  Some  experience 

c.  No  experience,  company  is  start-up  and  situation  is  highly  dynamic 

10.  How  would  you  describe  the  operational  control  of  the  organization  affected  by 
the  COTS  product(s)  implementation? 

a.  Centralized 

b.  Combination  of  centralized  and  decentralized 

c.  Decentralized 
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1 1 .  How  would  you  describe  the  sufficiency  of  skilled  staff  in  the  system/program 
affected  by  the  COTS  application  package(s)  implementation? 

a.  Sufficiently  staffed  and  skilled 

b.  Minimally  staffed  and  skilled 

c.  Insufficiently  staffed  and  skilled 

12.  How  much  experience  does  the  COTS  implementation  project  team  have  with  the 
COTS  product(s)? 

a.  Extensive  experience 

b.  Some  experience 

c.  No  experience 

13.  How  much  experience  does  the  project  team  have  with  implementation  of  other 
COTS  products? 

a.  Experienced  with  many  COTS  products 

b.  Experienced  with  a  few  COTS  products 

c.  Experienced  with  no  other  COTS  products 

14.  What  is  the  vendor's  track  record  with  implementing  the  COTS  product(s)  within 
their  cost  proposal? 

a.  Below  total  life  cycle  cost  estimate 

b.  Met  total  life  cycle  cost  estimate 

c.  Exceeded  total  life  cycle  cost  estimate 

15.  How  financially  stable  is  the  vendor? 

a.  Solid  financial  situation 

b.  Mixed  financial  picture,  may  have  strong  revenue  but  no  profit  margin 

c.  Financial  problems,  such  as  poor  credit,  low  revenues  and  low  profit 
margin 

16.  To  what  extent  does  your  acquisition  approach  include  an  understanding  of  the 
vendor's  future  plans  for  the  COTS  product(s)? 

a.  Statement  of  direction  for  the  product,  including  planned  enhancements 
and  release  dates,  has  been  received 

b.  Discussions  have  been  conducted  with  vendor  regarding  future  direction, 
but  no  plans  have  been  received  in  writing 

c.  No  discussion  with  vendor  regarding  future  direction 
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17.  Have  data  rights  been  properly  negotiated  with  the  vendors? 

a.  All  data  rights  negotiated  into  the  contract 

b.  Many  data  rights  have  been  negotiated  into  the  contract 

c.  No  data  rights  have  been  negotiated  into  the  contract 

18.  Have  cost-effective  licensing  agreements  been  worked  out  with  the  vendors? 

a.  All  licensing  agreements  negotiated  into  cost 

b.  Many  licensing  agreements  negotiated  into  cost 

c.  Uncertain  what  licensing  agreements  are  needed 


Responses  in  Process  Section: 

#  a _ x  1  = _ 

#  b _ x  2  = _ 

#  c _ x  3  = _ 

Total  = 


Technology 

1 .  Is  the  COTS  application  package(s)  a  totally  new  system  for  the  organization? 

a.  System  is  a  replacement 

b.  Components  of  the  system  are  new 

c.  New  system 

2.  To  adequately  address  your  organization’s  needs,  what  is  the  level  of 
customization  required  for  the  COTS  product(s)  baseline? 

a.  No  customization  necessary 

b.  Some  customization  necessary 

c.  Much  customization  necessary 
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3.  How  do  the  COTS  application  package(s)  "fit"  with  the  organization's  existing 
and  planned  architecture? 

a.  Good  fit 

b.  May  fit 

c.  Not  a  fit 

4.  Is  the  COTS  product(s)  view  as  a  time-tested,  mature  product? 

a.  Very  mature 

b.  Somewhat  mature 

c.  New  or  immature 

5.  How  many  functions  (e.g.,  accounting,  procurement)  are  supported  by  the  COTS 
application  package(s)? 

a.  Single  function 

b.  Few  functions 

c.  Many  functions 

6.  How  would  you  describe  the  complexity  of  the  interfaces  between  the  COTS 
product(s)  and  other  systems? 

a.  Simple,  easy  to  understand 

b.  Somewhat  complex 

c.  Very  complex,  difficult 

7.  How  many  systems  interfaces  must  remain  unchanged  after  the  implementation  of 
the  COTS  product(s)? 

a.  Few 

b.  Some 

c.  Many 

8.  How  would  you  describe  the  sufficiency  of  documentation  supporting  the 
system(s)  with  which  the  COTS  application  package(s)  will  interface? 

a.  Extremely  high  quality,  thorough  documentation 

b.  Adequate,  some  documentation 

c.  Poor  documentation  or  does  not  exist 
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9.  To  what  extent  has  your  organization  tested  COTS  application  package(s)  in  your 
environment? 


a.  Conducted  extensive  testing 

b.  Conducted  some  testing 

c.  Have  not  conducted  any  testing 

10.  Do  the  security  features  included  in  the  COTS  product(s)  need  modification  to 
meet  your  organization's  requirements? 

a.  Meets  security  requirements,  no  modification  needed 

b.  Meets  most  security  requirements,  some  modification  needed 

c.  Will  not  handle  security  requirements,  extensive  modification  needed 

1 1 .  How  flexible  is  the  design  of  the  COTS  product(s)  to  allow  for  future  changes  in 
functionality? 

a.  Very  flexible — product  functions  can  be  easily  separated  to  be  modified 

b.  Moderately  flexible — product  functions  can  be  separated  to  be  modified 

c.  Not  flexible — product  functions  can  not  be  separated  to  be  modified 

12.  What  is  the  reliability  history  of  the  COTS  product? 

a.  Product  is  stable  and  has  proven  itself  over  time  with  its  customer  base 

b.  Product  has  occasional  errors  but  none  will  result  in  data  loss  or  other 
critical  problem 

c.  Product  has  errors  that  result  in  data  loss,  work  lost,  system  crashes,  etc. 


Responses  in  Technology  Section: 

#  a _ x  1  = _ 

#  b _ x  2  = _ 

#  c _ x  3  = _ 

Total  = 
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Implementation/Logistics  Support 

1.  Has  your  organization  examined  and  applied  the  lessons  learned  from  other 
organizations  that  implemented  the  COTS  application  package(s)? 

a.  Yes — relevant  lessons  learned  have  been  incorporated  into  the 
implementation  plan 

b.  Somewhat — past  projects  have  been  discussed  by  the  project  team 

c.  No — have  not  gathered  any  infonnation  regarding  other  implementations 

2.  How  will  your  organization  measure  the  impact  and  effectiveness  of  the  COTS 
product(s)? 

a.  Comprehensive  performance  measures  (including  cost,  time  spent  on  each 
activity,  etc.)  have  been  established 

b.  Performance  measures  have  been  discussed  but  not  finalized 

c.  No  discussion  of  performance  measures 

3.  What  sort  of  testing  approach  is  planned  for  the  COTS  product(s)? 

a.  Designed  specifically  for  a  COTS  implementation 

b.  Combines  traditional  systems  development  testing  with  COTS-specific 
testing 

c.  Designed  for  traditional  systems  development  activities 

4.  How  would  you  describe  your  organization's  ability  to  support  new  releases  of  the 
COTS  product(s)? 

a.  Sufficient — staffing  plan  for  ongoing  support  of  the  COTS  application 
package(s)  has  been  developed 

b.  Moderate — staffing  needs  have  been  identified,  but  plan  has  not  been 
finalized 

c.  Minimal — no  staff  resources  are  available  after  the  initial  implementation 

5.  How  has  the  organization  prepared  for  the  possibility  that  the  COTS  application 
package(s)  vendor  goes  out  of  business  or  discontinues  support  for  the  product? 

a.  Contingency  plan  finalized  and  ready  to  implement 

b.  Possibility  discussed,  but  have  no  finalized  plan 

c.  Possibility  not  discussed,  no  contingency  plan  being  developed 
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6.  How  would  you  describe  the  run  time  perfonnance  of  the  COTS  product(s)  in 
your  environment? 

a.  Very  efficient 

b.  Moderately  efficient 

c.  Not  efficient 

7.  Does  the  run  time  perfonnance  of  the  COTS  application  package(s)  meet  the 
organization's  performance  needs? 

a.  Efficiently  supports  the  number  and  location  of  users 

b.  Supports  needs  with  performance  degradation 

c.  Does  not  support  needs 

8.  How  do  other  users  of  the  COTS  product  describe  their  satisfaction  with 
availability  of  the  vendor  staff? 

a.  Very  satisfied,  easy  to  access  key  personnel  at  vendor 

b.  Somewhat  satisfied,  can  access  key  personnel  some  of  the  time 

c.  Unsatisfied,  access  to  key  personnel  is  difficult 

9.  What  training  is  needed  to  operate  and  maintain  the  COTS  product? 

a.  No  training 

b.  Some  training 

c.  Extensive  training 

7.  What  training  sources  are  available  to  the  customers? 

a.  Extensive  training  resources 

b.  Some  training  resources 

c.  No  training  resources 

1 1 .  How  much  experience  does  other  support  contractors  serving  your  organization  in 
functions  affected  by  the  COTS  implementation  have  with  the  COTS  application 
package(s)? 

a. 

b. 

c. 


Extensive  experience 
Some  experience 
No  experience 


APPENDIX  C  DLA  BUSINESS  SYSTEMS  MODERNIZATION 
QUESTIONNAIRE  RESPONSES 


SECTION  I 

Service:  Defense  Logistics  Agency _ 

Agency  or  Organization: _ Defense  Logistics  Agency _ 

Project/System  Name:  Business  Systems  Modernization _ 

Which  category  best  describes  your  main  job  function  in  your  organization? 

a.  Management 

a.  System  analysis  or  design 

b.  Application  or  system  programming 

Have  you  participated  in  selecting  COTS  software  components  that  where  later 
adapted  or  integrated  into  your  project/system?  How  many  times?  First  ERP  -  Other 
minor  COTS  projects  in  past  -  never  of  this  magnitude  (enterprise- wide). 

How  long  is  your  work  experience  with  building  system  from  COTS 
components?  10  Years 

Please  state  any  good  practices,  or  lessons  you  have  learnt  from  past  experience 
when  acquiring  and  developing  systems  using  COTS  software. 

•  Do  not  modify  core  COTS  software 

•  Basic  integration  practices  for  COTS  are  the  same  as  software 
development  (i.e.,  basics  of  configuration  management,  software  QA, 
repeatable  processes,  etc. 

•  Willingness  to  adapt 

•  Completeness  of  requirements 


SECTION  II 

Questions  are  organized  around  the  three  broad  areas  of  implementing  a  COTS 
solution  as  presented  above.  Each  question  prompts  you,  the  respondent,  to  think  about 
key  factors  for  a  successful  COTS  application  package  implementation.  You  should 
carefully  consider  your  answer  in  terms  of  how  it  pertains  to  projects  within  your  own 
organization. 
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Answers  to  each  question  are  provided  by  the  choice  a,  b  or  c,  which  correlate  to 
the  three  levels  of  risk:  low,  medium  and  high,  respectively.  Please  record  your  answers 
to  the  questionnaire  directly  on  this  form.  We  ask  that  you  make  the  best  effort  possible 
to  provide  an  answer  to  all  the  questions.  If  you  are  unsure  of  an  answer,  or  feel  a 
question  does  not  apply  to  your  project,  please  indicate  so  rather  than  leaving  a  question 
blank. 

Process 

1 .  How  well  are  the  requirements  for  your  system/program  documented? 

a.  Thoroughly — comprehensive,  current  documentation  exists 

b.  Moderately  well — comprehensive  documentation  exists,  but  has  not 
been  updated  recently 

c.  Poorly — minimal  documentation  exists 

2.  Because  specific  requirements  are  associated  with  each  COTS  application 
package(s),  how  would  you  describe  the  relationship  between  the  specifications  of  the 
COTS  product(s)  and  the  requirements  of  your  system/program? 

a.  Ideal — great  fit,  fully  meets  requirements 

b.  Satisfactory — acceptable  fit,  meets  most  requirements 

c.  Unsatisfactory — marginal  fit,  must  be  modified  to  meet  requirements 

3.  How  many  COTS  product(s)  can  accommodate  your  system/program 
requirements? 

a.  Many 

b.  Some 

c.  Few 
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4.  How  would  you  describe  the  process  by  which  your  organization  will  implement 
new  requirements  after  the  initial  implementation  of  the  COTS  product(s)? 

a.  Well-defined,  proven  process  has  been  established  to  evaluate  and 
implement  new  requirements  (e.g.,  configuration  control  board) 

b.  Process  for  evaluating  and  implementing  new  requirements  has  been 
discussed,  but  not  solidified 

c.  No  process  exists  for  evaluating  and  implementing  new  requirements 


5.  How  would  you  describe  your  system/program’s  ability  to  adapt  to  the  new 
requirements  supported  by  the  COTS  product(s)? 

a.  Very  able — there  is  a  general  understanding  that  the  new 
requirements  would  enhance  organization's  operation 

b.  Somewhat  able — there  is  a  general  understanding  that  the  new 
requirements  would  not  enhance  or  deter  organization's  operation 

c.  Not  able — there  is  a  general  understanding  that  the  new  requirements 
would  deter  organization’s  operation 


6.  How  was  the  COTS  product  evaluated  and  selected? 

a.  Well-defined,  proven  process  has  been  established  to  evaluate  and 
select  cots  product 

b.  Process  for  evaluating  and  selecting  COTS  products  have  been  discussed, 
but  not  solidified 

c.  Ad  hoc,  no  process  exists  for  evaluating  and  implementing  new 
requirements 


7.  What  is  the  vendor's  experience  with  implementing  the  COTS  product(s)  in 
organizations  of  a  size  similar  to  yours? 

a.  Extensive  experience,  established  company  with  quality  workforce 
and  facilities 

b.  Some  experience 

c.  No  experience,  company  is  start-up  and  situation  is  highly  dynamic 
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8.  How  has  the  vendor  performed  in  the  integration  of  the  COTS  application 
package(s)  elsewhere? 

a.  Excellent  past  performance 

b.  Good  past  performance 

c.  Poor  or  unknown  past  performance 


9.  What  is  the  vendor's  experience  with  implementing  the  considered  COTS 
product(s)  in  organizations  of  a  management  structure  similar  to  yours? 

a.  Extensive  experience,  established  company  with  quality  workforce  and 
facilities 

b.  Some  experience  -  first  major  DOD  implementation 

c.  No  experience,  company  is  start-up  and  situation  is  highly  dynamic 


10.  How  would  you  describe  the  operational  control  of  the  organization  affected  by 
the  COTS  product(s)  implementation? 


a.  Centralized 

b.  Combination  of  centralized  and  decentralized 

c.  Decentralized 


1 1 .  How  would  you  describe  the  sufficiency  of  skilled  staff  in  the  system/program 
affected  by  the  COTS  application  package(s)  implementation? 

a.  Sufficiently  staffed  and  skilled 

b.  Minimally  staffed  and  skilled 

c.  Insufficiently  staffed  and  skilled 
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12.  How  much  experience  does  the  COTS  implementation  project  team  have  with  the 
COTS  product(s)? 

a.  Extensive  experience 

b.  Some  experience 

c.  No  experience 

13.  How  much  experience  does  the  project  team  have  with  implementation  of  other 
COTS  products? 

Experienced  with  many  COTS  products 

Experienced  with  a  few  COTS  products 

Experienced  with  no  other  COTS  products 

14.  What  is  the  vendor's  track  record  with  implementing  the  COTS  product(s)  within 
their  cost  proposal? 

a.  Below  total  life  cycle  cost  estimate 

b.  Met  total  life  cycle  cost  estimate 

c.  Exceeded  total  life  cycle  cost  estimate 


15.  How  financially  stable  is  the  vendor? 

a.  Solid  financial  situation 

b.  Mixed  financial  picture,  may  have  strong  revenue  but  no  profit  margin 

c.  Financial  problems,  such  as  poor  credit,  low  revenues  and  low  profit 
margin 
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16.  To  what  extent  does  your  acquisition  approach  include  an  understanding  of  the 
vendor's  future  plans  for  the  COTS  product(s)? 

a.  Statement  of  direction  for  the  product,  including  planned 
enhancements  and  release  dates,  has  been  received 

b.  Discussions  have  been  conducted  with  vendor  regarding  future  direction, 
but  no  plans  have  been  received  in  writing 

c.  No  discussion  with  vendor  regarding  future  direction 

17.  Have  data  rights  been  properly  negotiated  with  the  vendors? 

a.  All  data  rights  negotiated  into  the  contract 

b.  Many  data  rights  have  been  negotiated  into  the  contract 

c.  No  data  rights  have  been  negotiated  into  the  contract 

18.  Have  cost-effective  licensing  agreements  been  worked  out  with  the  vendors? 

a.  All  licensing  agreements  negotiated  into  cost 

b.  Many  licensing  agreements  negotiated  into  cost 

c.  Uncertain  what  licensing  agreements  are  needed 


Responses  in  Process  Section: 

#  a  _1_2_  x  1  =  J2 

#  b  6_  x  2  =  _12 

#  c  0_  x  3  =  _0 

Total  =  24 


Technology 

1 .  Is  the  COTS  application  package(s)  a  totally  new  system  for  the  organization? 

a.  System  is  a  replacement 

b.  Components  of  the  system  are  new 

c.  New  system 
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2.  To  adequately  address  your  organization’s  needs,  what  is  the  level  of 
customization  required  for  the  COTS  product(s)  baseline? 

a.  No  customization  necessary 

b.  Some  customization  necessary 

c.  Much  customization  necessary 

3.  How  do  the  COTS  application  package(s)  "fit"  with  the  organization's  existing 
and  planned  architecture? 

a.  Good  fit 

b.  May  fit 

c.  Not  a  fit 


4.  Is  the  COTS  product(s)  view  as  a  time-tested,  mature  product? 

a.  Very  mature 

b.  Somewhat  mature 

c.  New  or  immature 


5.  How  many  functions  (e.g.,  accounting,  procurement)  are  supported  by  the  COTS 
application  package(s)? 

a.  Single  function 

b.  Few  functions 

c.  Many  functions 


6.  How  would  you  describe  the  complexity  of  the  interfaces  between  the  COTS 
product(s)  and  other  systems? 

a.  Simple,  easy  to  understand 

b.  Somewhat  complex 

Very  complex,  difficult 


c. 


97 


7.  How  many  systems  interfaces  must  remain  unchanged  after  the  implementation  of 
the  COTS  product(s)? 


a.  Few 

b.  Some 

c.  Many 


8.  How  would  you  describe  the  sufficiency  of  documentation  supporting  the 
system(s)  with  which  the  COTS  application  package(s)  will  interface? 

a.  Extremely  high  quality,  thorough  documentation 

b.  Adequate,  some  documentation 

c.  Poor  documentation  or  does  not  exist 


9.  To  what  extent  has  your  organization  tested  COTS  application  package(s)  in  your 
environment? 

a.  Conducted  extensive  testing 

b.  Conducted  some  testing 

c.  Have  not  conducted  any  testing 

10.  Do  the  security  features  included  in  the  COTS  product(s)  need  modification  to 
meet  your  organization's  requirements? 

a.  Meets  security  requirements,  no  modification  needed 

b.  Meets  most  security  requirements,  some  modification  needed 

c.  Will  not  handle  security  requirements,  extensive  modification  needed 
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1 1 .  How  flexible  is  the  design  of  the  COTS  product(s)  to  allow  for  future  changes  in 
functionality? 

a.  Very  flexible — product  functions  can  be  easily  separated  to  be  modified 

b.  Moderately  flexible — product  functions  can  be  separated  to  be 
modified 

c.  Not  flexible — product  functions  can  not  be  separated  to  be  modified 


12.  What  is  the  reliability  history  of  the  COTS  product? 

a.  Product  is  stable  and  has  proven  itself  over  time  with  its  customer 
base 

b.  Product  has  occasional  errors  but  none  will  result  in  data  loss  or  other 
critical  problem 

c.  Product  has  errors  that  result  in  data  loss,  work  lost,  system  crashes,  etc. 


Responses  in  Technology  Section: 

#a^_x1  =  _5 

#  b  _3_  x  2  =  _6 

#  c  _4_  x  3  =  J_2 

Total  =  23 


Implementation/Logistics  Support 

1.  Has  your  organization  examined  and  applied  the  lessons  learned  from  other 
organizations  that  implemented  the  COTS  application  package(s)? 

a.  Yes — relevant  lessons  learned  have  been  incorporated  into  the 
implementation  plan 

b.  Somewhat — past  projects  have  been  discussed  by  the  project  team 

c.  No — have  not  gathered  any  infonnation  regarding  other  implementations 
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2.  How  will  your  organization  measure  the  impact  and  effectiveness  of  the  COTS 
product(s)? 

a.  Comprehensive  performance  measures  (including  cost,  time  spent  on 
each  activity,  etc.)  have  been  established 

b.  Performance  measures  have  been  discussed  but  not  finalized 

c.  No  discussion  of  performance  measures 

3.  What  sort  of  testing  approach  is  planned  for  the  COTS  product(s)? 

a.  Designed  specifically  for  a  COTS  implementation 

b.  Combines  traditional  systems  development  testing  with  COTS-specific 
testing 

c.  Designed  for  traditional  systems  development  activities 

4.  How  would  you  describe  your  organization's  ability  to  support  new  releases  of  the 
COTS  product(s)? 

a.  Sufficient — staffing  plan  for  ongoing  support  of  the  COTS  application 
package(s)  has  been  developed 

b.  Moderate — staffing  needs  have  been  identified,  but  plan  has  not  been 
finalized 

c.  Minimal — no  staff  resources  are  available  after  the  initial  implementation 


5.  How  has  the  organization  prepared  for  the  possibility  that  the  COTS  application 
package(s)  vendor  goes  out  of  business  or  discontinues  support  for  the  product? 

a.  Contingency  plan  finalized  and  ready  to  implement 

b.  Possibility  discussed,  but  have  no  finalized  plan 

c.  Possibility  not  discussed,  no  contingency  plan  being  developed 
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6.  How  would  you  describe  the  run  time  perfonnance  of  the  COTS  product(s)  in 
your  environment? 

a.  Very  efficient 

b.  Moderately  efficient 

c.  Not  efficient 


7.  Does  the  run  time  performance  of  the  COTS  application  package(s)  meet  the 
organization's  performance  needs? 

a.  Efficiently  supports  the  number  and  location  of  users 

b.  Supports  needs  with  performance  degradation 

c.  Does  not  support  needs 


8.  How  do  other  users  of  the  COTS  product  describe  their  satisfaction  with 
availability  of  the  vendor  staff? 

a.  Very  satisfied,  easy  to  access  key  personnel  at  vendor 

b.  Somewhat  satisfied,  can  access  key  personnel  some  of  the  time 

c.  Unsatisfied,  access  to  key  personnel  is  difficult 


9.  What  training  is  needed  to  operate  and  maintain  the  COTS  product? 

a.  No  training 

b.  Some  training 

c.  Extensive  training 


10.  What  training  sources  are  available  to  the  customers? 

a.  Extensive  training  resources 

b.  Some  training  resources 

c.  No  training  resources 
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1 1 .  How  much  experience  does  other  support  contractors  serving  your  organization  in 
functions  affected  by  the  COTS  implementation  have  with  the  COTS  application 
package(s)? 

a.  Extensive  experience 

b.  Some  experience 

c.  No  experience 

Responses  in  Implementation/Logistical  Support  Section: 

#  a  _5_x  1  =  _5 

#  b  1x2  =  JO 

#  c  J_  x  3  =  _3 

Total  =  18 
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APPENDIX  D  ARMY  HUMAN  RESOURCE  SYSTEM 
QUESTIONNAIRE  RESPONSES 


SECTION  I 

Service:  U.S.  Army 

Agency  or  Organization: _ PEO  EIS,  AHRS 

Project/System  Name:  Army  Human  Resource  System 

Which  category  best  describes  your  main  job  function  in  your  organization? 

a.  Management 

a.  System  analysis  or  design 

b.  Application  or  system  programming 

Have  you  participated  in  selecting  COTS  software  components  that  where  later 
adapted  or  integrated  into  your  project/system?  How  many  times?  Yes,  at  least  20. 

How  long  is  your  work  experience  with  building  system  from  COTS 
components?  30  Years 

Please  state  any  good  practices,  or  lessons  you  have  learnt  from  past  experience 
when  acquiring  and  developing  systems  using  COTS  software. 

•  Do  not  use  software  that  does  not  have  a  long-standing  commercial  user 
base 

•  Never  be  forced  to  use  products  with  questionable  long-tenn  life-cycle 
support 

•  Do  not  allow  GOTS  products  to  be  forced  on  your  program,  these  are 
generally  built  with  COTS  products  no  longer  in  business. 


SECTION  II 

Questions  are  organized  around  the  three  broad  areas  of  implementing  a  COTS 
solution  as  presented  above.  Each  question  prompts  you,  the  respondent,  to  think  about 
key  factors  for  a  successful  COTS  application  package  implementation.  You  should 
carefully  consider  your  answer  in  terms  of  how  it  pertains  to  projects  within  your  own 
organization. 
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Answers  to  each  question  are  provided  by  the  choice  a,  b  or  c,  which  correlate  to 
the  three  levels  of  risk:  low,  medium  and  high,  respectively.  Please  record  your  answers 
to  the  questionnaire  directly  on  this  form.  We  ask  that  you  make  the  best  effort  possible 
to  provide  an  answer  to  all  the  questions.  If  you  are  unsure  of  an  answer,  or  feel  a 
question  does  not  apply  to  your  project,  please  indicate  so  rather  than  leaving  a  question 
blank. 

Process 

1 .  How  well  are  the  requirements  for  your  system/program  documented? 

a.  Thoroughly — comprehensive,  current  documentation  exists 

b.  Moderately  well — comprehensive  documentation  exists,  but  has  not  been 
updated  recently 

c.  Poorly — minimal  documentation  exists 

2.  Because  specific  requirements  are  associated  with  each  COTS  application 
package(s),  how  would  you  describe  the  relationship  between  the  specifications  of  the 
COTS  product(s)  and  the  requirements  of  your  system/program? 

a.  Ideal — great  fit,  fully  meets  requirements 

b.  Satisfactory — acceptable  fit,  meets  most  requirements 

c.  Unsatisfactory — marginal  fit,  must  be  modified  to  meet  requirements 

3.  How  many  COTS  product(s)  can  accommodate  your  system/program 
requirements? 

a.  Many 

b.  Some 

c.  Few 
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4.  How  would  you  describe  the  process  by  which  your  organization  will  implement 
new  requirements  after  the  initial  implementation  of  the  COTS  product(s)? 

a.  Well-defined,  proven  process  has  been  established  to  evaluate  and 
implement  new  requirements  (e.g.,  configuration  control  board) 

b.  Process  for  evaluating  and  implementing  new  requirements  has  been 
discussed,  but  not  solidified 

c.  No  process  exists  for  evaluating  and  implementing  new  requirements 


5.  How  would  you  describe  your  system/program’s  ability  to  adapt  to  the  new 
requirements  supported  by  the  COTS  product(s)? 

a.  Very  able — there  is  a  general  understanding  that  the  new 
requirements  would  enhance  organization's  operation 

b.  Somewhat  able — there  is  a  general  understanding  that  the  new 
requirements  would  not  enhance  or  deter  organization's  operation 

c.  Not  able — there  is  a  general  understanding  that  the  new  requirements 
would  deter  organization’s  operation 


6.  How  was  the  COTS  product  evaluated  and  selected? 

a.  Well-defined,  proven  process  has  been  established  to  evaluate  and 
select  cots  product 

b.  Process  for  evaluating  and  selecting  COTS  products  have  been  discussed, 
but  not  solidified 

c.  Ad  hoc,  no  process  exists  for  evaluating  and  implementing  new 
requirements 


7.  What  is  the  vendor's  experience  with  implementing  the  COTS  product(s)  in 
organizations  of  a  size  similar  to  yours? 

a.  Extensive  experience,  established  company  with  quality  workforce 
and  facilities 

b.  Some  experience 

c.  No  experience,  company  is  start-up  and  situation  is  highly  dynamic 
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8.  How  has  the  vendor  performed  in  the  integration  of  the  COTS  application 
package(s)  elsewhere? 

a.  Excellent  past  performance 

b.  Good  past  performance 

c.  Poor  or  unknown  past  performance 


9.  What  is  the  vendor's  experience  with  implementing  the  considered  COTS 
product(s)  in  organizations  of  a  management  structure  similar  to  yours? 

a.  Extensive  experience,  established  company  with  quality  workforce 
and  facilities 

b.  Some  experience  -  first  major  DOD  implementation 

c.  No  experience,  company  is  start-up  and  situation  is  highly  dynamic 


10.  How  would  you  describe  the  operational  control  of  the  organization  affected  by 
the  COTS  product(s)  implementation? 

a.  Centralized 

b.  Combination  of  centralized  and  decentralized 

c.  Decentralized 


1 1 .  How  would  you  describe  the  sufficiency  of  skilled  staff  in  the  system/program 
affected  by  the  COTS  application  package(s)  implementation? 

a.  Sufficiently  staffed  and  skilled 

b.  Minimally  staffed  and  skilled 

c.  Insufficiently  staffed  and  skilled 
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12.  How  much  experience  does  the  COTS  implementation  project  team  have  with  the 
COTS  product(s)? 

a.  Extensive  experience 

b.  Some  experience 

c.  No  experience 

13.  How  much  experience  does  the  project  team  have  with  implementation  of  other 
COTS  products? 

a.  Experienced  with  many  COTS  products 

b.  Experienced  with  a  few  COTS  products 

c.  Experienced  with  no  other  COTS  products 

14.  What  is  the  vendor's  track  record  with  implementing  the  COTS  product(s)  within 
their  cost  proposal? 

a.  Below  total  life  cycle  cost  estimate 

b.  Met  total  life  cycle  cost  estimate 

c.  Exceeded  total  life  cycle  cost  estimate 


15.  How  financially  stable  is  the  vendor? 

a.  Solid  financial  situation 

b.  Mixed  financial  picture,  may  have  strong  revenue  but  no  profit  margin 

c.  Financial  problems,  such  as  poor  credit,  low  revenues  and  low  profit 
margin 
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16.  To  what  extent  does  your  acquisition  approach  include  an  understanding  of  the 
vendor's  future  plans  for  the  COTS  product(s)? 

a.  Statement  of  direction  for  the  product,  including  planned 
enhancements  and  release  dates,  has  been  received 

b.  Discussions  have  been  conducted  with  vendor  regarding  future  direction, 
but  no  plans  have  been  received  in  writing 

c.  No  discussion  with  vendor  regarding  future  direction 

17.  Have  data  rights  been  properly  negotiated  with  the  vendors? 

a.  All  data  rights  negotiated  into  the  contract 

b.  Many  data  rights  have  been  negotiated  into  the  contract 

c.  No  data  rights  have  been  negotiated  into  the  contract 

18.  Have  cost-effective  licensing  agreements  been  worked  out  with  the  vendors? 

a.  All  licensing  agreements  negotiated  into  cost 

b.  Many  licensing  agreements  negotiated  into  cost 

c.  Uncertain  what  licensing  agreements  are  needed 


Responses  in  Process  Section: 

#  a  _Y7_  x  1  =J7 

#  b  1x2  =  _2 

#  c  0_  x  3  =  _0 

Total  =  19 
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Technology 

1 .  Is  the  COTS  application  package(s)  a  totally  new  system  for  the  organization? 

a.  System  is  a  replacement 

b.  Components  of  the  system  are  new 

c.  New  system 

2.  To  adequately  address  your  organization’s  needs,  what  is  the  level  of 
customization  required  for  the  COTS  product(s)  baseline? 

a.  No  customization  necessary 

b.  Some  customization  necessary 

c.  Much  customization  necessary 

3.  How  do  the  COTS  application  package(s)  "fit"  with  the  organization's  existing 
and  planned  architecture? 

a.  Good  fit 

b.  May  fit 

c.  Not  a  fit 

4.  Is  the  COTS  product(s)  view  as  a  time-tested,  mature  product? 

a.  Very  mature 

b.  Somewhat  mature 

c.  New  or  immature 

5.  How  many  functions  (e.g.,  accounting,  procurement)  are  supported  by  the  COTS 
application  package(s)? 

a.  Single  function 

b.  Few  functions 

c.  Many  functions 
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6.  How  would  you  describe  the  complexity  of  the  interfaces  between  the  COTS 
product(s)  and  other  systems? 

a.  Simple,  easy  to  understand 

b.  Somewhat  complex 

c.  Very  complex,  difficult 


7.  How  many  systems  interfaces  must  remain  unchanged  after  the  implementation  of 
the  COTS  product(s)? 

a.  Few 

b.  Some 

c.  Many 


8.  How  would  you  describe  the  sufficiency  of  documentation  supporting  the 
system(s)  with  which  the  COTS  application  package(s)  will  interface? 

a.  Extremely  high  quality,  thorough  documentation 

b.  Adequate,  some  documentation 

c.  Poor  documentation  or  does  not  exist 


9.  To  what  extent  has  your  organization  tested  COTS  application  package(s)  in  your 
environment? 

a.  Conducted  extensive  testing 

b.  Conducted  some  testing 

c.  Have  not  conducted  any  testing 
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10.  Do  the  security  features  included  in  the  COTS  product(s)  need  modification  to 
meet  your  organization's  requirements? 

a.  Meets  security  requirements,  no  modification  needed 

b.  Meets  most  security  requirements,  some  modification  needed 

c.  Will  not  handle  security  requirements,  extensive  modification  needed 


1 1 .  How  flexible  is  the  design  of  the  COTS  product(s)  to  allow  for  future  changes  in 
functionality? 

a.  Very  flexible — product  functions  can  be  easily  separated  to  be 
modified 

b.  Moderately  flexible — product  functions  can  be  separated  to  be  modified 

c.  Not  flexible — product  functions  can  not  be  separated  to  be  modified 


12.  What  is  the  reliability  history  of  the  COTS  product? 

a.  Product  is  stable  and  has  proven  itself  over  time  with  its  customer 
base 

b.  Product  has  occasional  errors  but  none  will  result  in  data  loss  or  other 
critical  problem 

c.  Product  has  errors  that  result  in  data  loss,  work  lost,  system  crashes,  etc. 


Responses  in  Technology  Section: 

#  a  7_  x  1  =  J_ 

#b  _3_  x  2  =  _6 

#  c  _2_  x  3  =  _6 

Total  =  19 
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Implementation/Logistics  Support 


1.  Has  your  organization  examined  and  applied  the  lessons  learned  from  other 
organizations  that  implemented  the  COTS  application  package(s)? 

a.  Yes — relevant  lessons  learned  have  been  incorporated  into  the 
implementation  plan 

b.  Somewhat — past  projects  have  been  discussed  by  the  project  team 

c.  No — have  not  gathered  any  infonnation  regarding  other  implementations 


2.  How  will  your  organization  measure  the  impact  and  effectiveness  of  the  COTS 
product(s)? 

a.  Comprehensive  perfonnance  measures  (including  cost,  time  spent  on 
each  activity,  etc.)  have  been  established 

b.  Performance  measures  have  been  discussed  but  not  finalized 

c.  No  discussion  of  performance  measures 


3.  What  sort  of  testing  approach  is  planned  for  the  COTS  product(s)? 

a.  Designed  specifically  for  a  COTS  implementation 

b.  Combines  traditional  systems  development  testing  with  COTS-specific 
testing 

c.  Designed  for  traditional  systems  development  activities 

4.  How  would  you  describe  your  organization's  ability  to  support  new  releases  of  the 
COTS  product(s)? 

a.  Sufficient — been  developed 

b.  Moderate — staffing  needs  have  been  identified,  but  plan  has  not  been 
finalized 

c.  Minimal — no  staff  resources  are  available  after  the  initial  implementation 
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5.  How  has  the  organization  prepared  for  the  possibility  that  the  COTS  application 
package(s)  vendor  goes  out  of  business  or  discontinues  support  for  the  product? 

a.  Contingency  plan  finalized  and  ready  to  implement 

b.  Possibility  discussed,  but  have  no  finalized  plan 

c.  Possibility  not  discussed,  no  contingency  plan  being  developed 

6.  How  would  you  describe  the  run  time  perfonnance  of  the  COTS  product(s)  in 
your  environment? 

a.  Very  efficient 

b.  Moderately  efficient 

c.  Not  efficient 

7.  Does  the  run  time  perfonnance  of  the  COTS  application  package(s)  meet  the 
organization's  performance  needs? 

a.  Efficiently  supports  the  number  and  location  of  users 

b.  Supports  needs  with  performance  degradation 

c.  Does  not  support  needs 

8.  How  do  other  users  of  the  COTS  product  describe  their  satisfaction  with 
availability  of  the  vendor  staff? 

a.  Very  satisfied,  easy  to  access  key  personnel  at  vendor 

b.  Somewhat  satisfied,  can  access  key  personnel  some  of  the  time 

c.  Unsatisfied,  access  to  key  personnel  is  difficult 

9.  What  training  is  needed  to  operate  and  maintain  the  COTS  product? 

a.  No  training 

b.  Some  training 

c.  Extensive  training 
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10.  What  training  sources  are  available  to  the  customers? 

a.  Extensive  training  resources 

b.  Some  training  resources 

c.  No  training  resources 

1 1 .  How  much  experience  does  other  support  contractors  serving  your  organization  in 
functions  affected  by  the  COTS  implementation  have  with  the  COTS  application 
package(s)? 

a.  Extensive  experience 

b.  Some  experience 

c.  No  experience 

Responses  in  Implementation/Logistical  Support  Section: 

#  a  _6_  x  1  =  _6 

#  b  _4_  x  2  =  _8 

#  c  J_  x  3  =  _3 

Total  =  17 


114 


APPENDIX  E  ARMY  COMMUNICATION  SOFTWARE  SUPPORT 
DIVIDION  QUESTIONNAIRE  RESPONSES 


SECTION  I 

Service:  U.S.  Army _ 

Agency  or  Organization: _ CECOM-SEC  or  AMSEL-SE-WS-COM 

Project/System  Name:  Post  Production  Software  Support 

Which  category  best  describes  your  main  job  function  in  your  organization? 

a.  Management  and  Software  Support 

b.  System  analysis  or  design 

c.  Application  or  system  programming 

Have  you  participated  in  selecting  COTS  software  components  that  where  later 
adapted  or  integrated  into  your  project/system?  How  many  times?  We  made 
recommendations  based  on  the  legality  use  and  when  COTS  products  are  no  longer 
supported  or  reach  end  of  life.  There  were  two  incidents  where  our 
recommendations  of  COTS  replacement  were  integrated. 

How  long  is  your  work  experience  with  building  system  from  COTS 
components?  4Years 

Please  state  any  good  practices,  or  lessons  you  have  learnt  from  past  experience 
when  acquiring  and  developing  systems  using  COTS  software. 

•  Know  your  requirements  well 

•  Assess  and  evaluate  different  available  COTS  products  and  its  cost  based 
on  the  requirements  way  in  advance 

•  Close,  continuous,  and  active  partnership  among  the  vendor,  customers, 
developer,  and  most  importantly  the  users 


SECTION  II 

Questions  are  organized  around  the  three  broad  areas  of  implementing  a  COTS 
solution  as  presented  above.  Each  question  prompts  you,  the  respondent,  to  think  about 
key  factors  for  a  successful  COTS  application  package  implementation.  You  should 
carefully  consider  your  answer  in  terms  of  how  it  pertains  to  projects  within  your  own 
organization. 
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Answers  to  each  question  are  provided  by  the  choice  a,  b  or  c,  which  correlate  to 
the  three  levels  of  risk:  low,  medium  and  high,  respectively.  Please  record  your  answers 
to  the  questionnaire  directly  on  this  form.  We  ask  that  you  make  the  best  effort  possible 
to  provide  an  answer  to  all  the  questions.  If  you  are  unsure  of  an  answer,  or  feel  a 
question  does  not  apply  to  your  project,  please  indicate  so  rather  than  leaving  a  question 
blank. 

Process 

1 .  How  well  are  the  requirements  for  your  system/program  documented? 

a.  Thoroughly — comprehensive,  current  documentation  exists 

b.  Moderately  well — comprehensive  documentation  exists,  but  has  not  been 
updated  recently 

c.  Poorly — minimal  documentation  exists 

2.  Because  specific  requirements  are  associated  with  each  COTS  application 
package(s),  how  would  you  describe  the  relationship  between  the  specifications  of  the 
COTS  product(s)  and  the  requirements  of  your  system/program? 

a.  Ideal — great  fit,  fully  meets  requirements 

b.  Satisfactory — acceptable  fit,  meets  most  requirements 

c.  Unsatisfactory — marginal  fit,  must  be  modified  to  meet  requirements 

3.  How  many  COTS  product(s)  can  accommodate  your  system/program 
requirements? 

a.  Many 

b.  Some 

c.  Few 


4.  How  would  you  describe  the  process  by  which  your  organization  will  implement 
new  requirements  after  the  initial  implementation  of  the  COTS  product(s)? 
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a.  Well-defined,  proven  process  has  been  established  to  evaluate  and 
implement  new  requirements  (e.g.,  configuration  control  board) 

b.  Process  for  evaluating  and  implementing  new  requirements  has  been 
discussed,  but  not  solidified 

c.  No  process  exists  for  evaluating  and  implementing  new  requirements 


5.  How  would  you  describe  your  system/program’s  ability  to  adapt  to  the  new 
requirements  supported  by  the  COTS  product(s)? 

a.  Very  able — there  is  a  general  understanding  that  the  new 
requirements  would  enhance  organization's  operation 

b.  Somewhat  able — there  is  a  general  understanding  that  the  new 
requirements  would  not  enhance  or  deter  organization's  operation 

c.  Not  able — there  is  a  general  understanding  that  the  new  requirements 
would  deter  organization's  operation 


6.  How  was  the  COTS  product  evaluated  and  selected? 

a.  Well-defined,  proven  process  has  been  established  to  evaluate  and  select 
cots  product 

b.  Process  for  evaluating  and  selecting  COTS  products  have  been 
discussed,  but  not  solidified 

c.  Ad  hoc,  no  process  exists  for  evaluating  and  implementing  new 
requirements 


7.  What  is  the  vendor's  experience  with  implementing  the  COTS  product(s)  in 
organizations  of  a  size  similar  to  yours? 

a.  Extensive  experience,  established  company  with  quality  workforce  and 
facilities 

b.  Some  experience 

c.  No  experience,  company  is  start-up  and  situation  is  highly  dynamic 
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8.  How  has  the  vendor  performed  in  the  integration  of  the  COTS  application 
package(s)  elsewhere? 

a.  Excellent  past  perfonnance 

b.  Good  past  performance 

c.  Poor  or  unknown  past  performance 

9.  What  is  the  vendor's  experience  with  implementing  the  considered  COTS 
product(s)  in  organizations  of  a  management  structure  similar  to  yours? 

a.  Extensive  experience,  established  company  with  quality  workforce  and 

facilities 

b.  Some  experience 

c.  No  experience,  company  is  start-up  and  situation  is  highly  dynamic 


10.  How  would  you  describe  the  operational  control  of  the  organization  affected  by 
the  COTS  product(s)  implementation? 

a.  Centralized 

b.  Combination  of  centralized  and  decentralized 

c.  Decentralized 


1 1 .  How  would  you  describe  the  sufficiency  of  skilled  staff  in  the  system/program 
affected  by  the  COTS  application  package(s)  implementation? 

a.  Sufficiently  staffed  and  skilled 

b.  Minimally  staffed  and  skilled 

c.  Insufficiently  staffed  and  skilled 
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12.  How  much  experience  does  the  COTS  implementation  project  team  have  with  the 
COTS  product(s)? 

a.  Extensive  experience 

b.  Some  experience 

c.  No  experience 

13.  How  much  experience  does  the  project  team  have  with  implementation  of  other 
COTS  products? 

a.  Experienced  with  many  COTS  products 

b.  Experienced  with  a  few  COTS  products 

c.  Experienced  with  no  other  COTS  products 

14.  What  is  the  vendor's  track  record  with  implementing  the  COTS  product(s)  within 
their  cost  proposal? 

a.  Below  total  life  cycle  cost  estimate 

b.  Met  total  life  cycle  cost  estimate 

c.  Exceeded  total  life  cycle  cost  estimate 

15.  How  financially  stable  is  the  vendor? 

a.  Solid  financial  situation 

b.  Mixed  financial  picture,  may  have  strong  revenue  but  no  profit 
margin 

c.  Financial  problems,  such  as  poor  credit,  low  revenues  and  low  profit 
margin 
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16.  To  what  extent  does  your  acquisition  approach  include  an  understanding  of  the 
vendor's  future  plans  for  the  COTS  product(s)? 

a.  Statement  of  direction  for  the  product,  including  planned  enhancements 
and  release  dates,  has  been  received 

b.  Discussions  have  been  conducted  with  vendor  regarding  future  direction, 
but  no  plans  have  been  received  in  writing 

c.  No  discussion  with  vendor  regarding  future  direction 

17.  Have  data  rights  been  properly  negotiated  with  the  vendors? 

a.  All  data  rights  negotiated  into  the  contract 

b.  Many  data  rights  have  been  negotiated  into  the  contract 

c.  No  data  rights  have  been  negotiated  into  the  contract 

18.  Have  cost-effective  licensing  agreements  been  worked  out  with  the  vendors? 

a.  All  licensing  agreements  negotiated  into  cost 

b.  Many  licensing  agreements  negotiated  into  cost 

c.  Uncertain  what  licensing  agreements  are  needed 

Responses  in  Process  Section: 

#  a  _3_  x  1  =_3 

#  b  12_x2  =  _24 

#  c  3_  x  3  =  _9 

Total  =  36 
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Technology 

1 .  Is  the  COTS  application  package(s)  a  totally  new  system  for  the  organization? 

a.  System  is  a  replacement 

b.  Components  of  the  system  are  new 

c.  New  system 

2.  To  adequately  address  your  organization’s  needs,  what  is  the  level  of 
customization  required  for  the  COTS  product(s)  baseline? 

a.  No  customization  necessary 

b.  Some  customization  necessary 

c.  Much  customization  necessary 

3.  How  do  the  COTS  application  package(s)  "fit"  with  the  organization's  existing 
and  planned  architecture? 

a.  Good  fit 

b.  May  fit 

c.  Not  a  fit 

4.  Is  the  COTS  product(s)  view  as  a  time-tested,  mature  product? 

a.  Very  mature 

b.  Somewhat  mature 

c.  New  or  immature 

5.  How  many  functions  (e.g.,  accounting,  procurement)  are  supported  by  the  COTS 
application  package(s)? 

a.  Single  function 

b.  Few  functions 

c.  Many  functions 
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6.  How  would  you  describe  the  complexity  of  the  interfaces  between  the  COTS 
product(s)  and  other  systems? 

a.  Simple,  easy  to  understand 

b.  Somewhat  complex 

c.  Very  complex,  difficult 


7.  How  many  systems  interfaces  must  remain  unchanged  after  the  implementation  of 
the  COTS  product(s)? 

a.  Few 

b.  Some 

c.  Many 


8.  How  would  you  describe  the  sufficiency  of  documentation  supporting  the 
system(s)  with  which  the  COTS  application  package(s)  will  interface? 

a.  Extremely  high  quality,  thorough  documentation 

b.  Adequate,  some  documentation 

c.  Poor  documentation  or  does  not  exist 


9.  To  what  extent  has  your  organization  tested  COTS  application  package(s)  in  your 
environment? 

a.  Conducted  extensive  testing 

b.  Conducted  some  testing 

c.  Have  not  conducted  any  testing 
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10.  Do  the  security  features  included  in  the  COTS  product(s)  need  modification  to 
meet  your  organization's  requirements? 

a.  Meets  security  requirements,  no  modification  needed 

b.  Meets  most  security  requirements,  some  modification  needed 

c.  Will  not  handle  security  requirements,  extensive  modification  needed 


1 1 .  How  flexible  is  the  design  of  the  COTS  product(s)  to  allow  for  future  changes  in 
functionality? 

a.  Very  flexible — product  functions  can  be  easily  separated  to  be  modified 

b.  Moderately  flexible — product  functions  can  be  separated  to  be  modified 

c.  Not  flexible — product  functions  can  not  be  separated  to  be  modified 


12.  What  is  the  reliability  history  of  the  COTS  product? 

a.  Product  is  stable  and  has  proven  itself  over  time  with  its  customer  base 

b.  Product  has  occasional  errors  but  none  will  result  in  data  loss  or  other 
critical  problem 

c.  Product  has  errors  that  result  in  data  loss,  work  lost,  system  crashes,  etc. 


Responses  in  Technology  Section: 

#  a  _2_  x  1  =2 

#  b  Jx2  =  J4 

#  c  _3_  x  3  =  _9 

Total  =  25 
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Implementation/Logistics  Support 


1.  Has  your  organization  examined  and  applied  the  lessons  learned  from  other 
organizations  that  implemented  the  COTS  application  package(s)? 

a.  Yes — relevant  lessons  learned  have  been  incorporated  into  the 
implementation  plan 

b.  Somewhat — past  projects  have  been  discussed  by  the  project  team 

c.  No — have  not  gathered  any  information  regarding  other 
implementations 

2.  How  will  your  organization  measure  the  impact  and  effectiveness  of  the  COTS 
product(s)? 

a.  Comprehensive  performance  measures  (including  cost,  time  spent  on  each 
activity,  etc.)  have  been  established 

b.  Performance  measures  have  been  discussed  but  not  finalized 

c.  No  discussion  of  performance  measures 

3.  What  sort  of  testing  approach  is  planned  for  the  COTS  product(s)? 

a.  Designed  specifically  for  a  COTS  implementation 

b.  Combines  traditional  systems  development  testing  with  COTS-specific 
testing 

c.  Designed  for  traditional  systems  development  activities 


4.  How  would  you  describe  your  organization's  ability  to  support  new  releases  of  the 
COTS  product(s)? 

a.  Sufficient — staffing  plan  for  ongoing  support  of  the  COTS  application 
package(s)  has  been  developed 

b.  Moderate — staffing  needs  have  been  identified,  but  plan  has  not  been 
finalized 

c.  Minimal — no  staff  resources  are  available  after  the  initial  implementation 
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5.  How  has  the  organization  prepared  for  the  possibility  that  the  COTS  application 
package(s)  vendor  goes  out  of  business  or  discontinues  support  for  the  product? 

a.  Contingency  plan  finalized  and  ready  to  implement 

b.  Possibility  discussed,  but  have  no  finalized  plan 

c.  Possibility  not  discussed,  no  contingency  plan  being  developed 

6.  How  would  you  describe  the  run  time  perfonnance  of  the  COTS  product(s)  in 
your  environment? 

a.  Very  efficient 

b.  Moderately  efficient 

c.  Not  efficient 

7.  Does  the  run  time  perfonnance  of  the  COTS  application  package(s)  meet  the 
organization's  performance  needs? 

a.  Efficiently  supports  the  number  and  location  of  users 

b.  Supports  needs  with  performance  degradation 

c.  Does  not  support  needs 

8.  How  do  other  users  of  the  COTS  product  describe  their  satisfaction  with 
availability  of  the  vendor  staff? 

a.  Very  satisfied,  easy  to  access  key  personnel  at  vendor 

b.  Somewhat  satisfied,  can  access  key  personnel  some  of  the  time 

c.  Unsatisfied,  access  to  key  personnel  is  difficult 

9.  What  training  is  needed  to  operate  and  maintain  the  COTS  product? 

a.  No  training 

b.  Some  training 

c.  Extensive  training 
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10.  What  training  sources  are  available  to  the  customers? 

a.  Extensive  training  resources 

b.  Some  training  resources 

c.  No  training  resources 

1 1 .  How  much  experience  does  other  support  contractors  serving  your  organization  in 
functions  affected  by  the  COTS  implementation  have  with  the  COTS  application 
package(s)? 

a.  Extensive  experience 

b.  Some  experience 

c.  No  experience 

Responses  in  Implementation/Logistical  Support  Section: 

#  a  _0  x  1  =  _5 

#  b  7_x2  =  J4 

#  c  _4_  x  3  =  _12 

Total  =  26 
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APPENDIX  F  ARMY  GLOBAL  COMBAT  SUPPORT  SYSTEM 
QUESTIONNAIRE  RESPONSES 


SECTION  I 

Service:  U.S.  Army 

Agency  or  Organization: _ PM  Logistics  Infonnation  Systems 

Project/System  Name:  Global  Combat  Support  System  -  Army  Tactical 

Which  category  best  describes  your  main  job  function  in  your  organization? 

a.  Management 

b.  System  analysis  or  design 

c.  Application  or  system  programming 

Have  you  participated  in  selecting  COTS  software  components  that  where  later 
adapted  or  integrated  into  your  project/system?  How  many  times?  Yes,  several 

How  long  is  your  work  experience  with  building  system  from  COTS 
components?  J_Years 

Please  state  any  good  practices,  or  lessons  you  have  learnt  from  past  experience 
when  acquiring  and  developing  systems  using  COTS  software. 

SECTION  II 

Questions  are  organized  around  the  three  broad  areas  of  implementing  a  COTS 
solution  as  presented  above.  Each  question  prompts  you,  the  respondent,  to  think  about 
key  factors  for  a  successful  COTS  application  package  implementation.  You  should 
carefully  consider  your  answer  in  terms  of  how  it  pertains  to  projects  within  your  own 
organization. 

Answers  to  each  question  are  provided  by  the  choice  a,  b  or  c,  which  correlate  to 
the  three  levels  of  risk:  low,  medium  and  high,  respectively.  Please  record  your  answers 
to  the  questionnaire  directly  on  this  form.  We  ask  that  you  make  the  best  effort  possible 
to  provide  an  answer  to  all  the  questions.  If  you  are  unsure  of  an  answer,  or  feel  a 
question  does  not  apply  to  your  project,  please  indicate  so  rather  than  leaving  a  question 
blank. 
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Process 


1 .  How  well  are  the  requirements  for  your  system/program  documented? 

a.  Thoroughly — comprehensive,  current  documentation  exists 

b.  Moderately  well — comprehensive  documentation  exists,  but  has  not  been 
updated  recently 

c.  Poorly — minimal  documentation  exists 

2.  Because  specific  requirements  are  associated  with  each  COTS  application 
package(s),  how  would  you  describe  the  relationship  between  the  specifications  of  the 
COTS  product(s)  and  the  requirements  of  your  system/program? 

a.  Ideal — great  fit,  fully  meets  requirements 

b.  Satisfactory — acceptable  fit,  meets  most  requirements 

c.  Unsatisfactory — marginal  fit,  must  be  modified  to  meet  requirements 

3.  How  many  COTS  product(s)  can  accommodate  your  system/program 
requirements? 

a.  Many 

b.  Some 

c.  Few 


4.  How  would  you  describe  the  process  by  which  your  organization  will  implement 
new  requirements  after  the  initial  implementation  of  the  COTS  product(s)? 

a.  Well-defined,  proven  process  has  been  established  to  evaluate  and 
implement  new  requirements  (e.g.,  configuration  control  board) 

b.  Process  for  evaluating  and  implementing  new  requirements  has  been 
discussed,  but  not  solidified 

c.  No  process  exists  for  evaluating  and  implementing  new  requirements 
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5.  How  would  you  describe  your  system/program’s  ability  to  adapt  to  the  new 
requirements  supported  by  the  COTS  product(s)? 

a.  Very  able — there  is  a  general  understanding  that  the  new 
requirements  would  enhance  organization's  operation 

b.  Somewhat  able — there  is  a  general  understanding  that  the  new 
requirements  would  not  enhance  or  deter  organization's  operation 

c.  Not  able — there  is  a  general  understanding  that  the  new  requirements 
would  deter  organization's  operation 


6.  How  was  the  COTS  product  evaluated  and  selected? 

a.  Well-defined,  proven  process  has  been  established  to  evaluate  and 
select  cots  product 

b.  Process  for  evaluating  and  selecting  COTS  products  have  been  discussed, 
but  not  solidified 

c.  Ad  hoc,  no  process  exists  for  evaluating  and  implementing  new 
requirements 


7.  What  is  the  vendor's  experience  with  implementing  the  COTS  product(s)  in 
organizations  of  a  size  similar  to  yours? 

a.  Extensive  experience,  established  company  with  quality  workforce 
and  facilities 

b.  Some  experience 

c.  No  experience,  company  is  start-up  and  situation  is  highly  dynamic 


8.  How  has  the  vendor  performed  in  the  integration  of  the  COTS  application 
package(s)  elsewhere? 

a.  Excellent  past  performance 

b.  Good  past  performance 

c.  Poor  or  unknown  past  performance 
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9.  What  is  the  vendor's  experience  with  implementing  the  considered  COTS 
product(s)  in  organizations  of  a  management  structure  similar  to  yours? 

a.  Extensive  experience,  established  company  with  quality  workforce 
and  facilities 

b.  Some  experience  -  first  major  DOD  implementation 

c.  No  experience,  company  is  start-up  and  situation  is  highly  dynamic 


10.  How  would  you  describe  the  operational  control  of  the  organization  affected  by 
the  COTS  product(s)  implementation? 

a.  Centralized 

b.  Combination  of  centralized  and  decentralized 

c.  Decentralized 


1 1 .  How  would  you  describe  the  sufficiency  of  skilled  staff  in  the  system/program 
affected  by  the  COTS  application  package(s)  implementation? 

a.  Sufficiently  staffed  and  skilled 

b.  Minimally  staffed  and  skilled 

c.  Insufficiently  staffed  and  skilled 

12.  How  much  experience  does  the  COTS  implementation  project  team  have  with  the 
COTS  product(s)? 

a.  Extensive  experience 

b.  Some  experience 

c.  No  experience 
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13.  How  much  experience  does  the  project  team  have  with  implementation  of  other 
COTS  products? 

a.  Experienced  with  many  COTS  products 

b.  Experienced  with  a  few  COTS  products 

c.  Experienced  with  no  other  COTS  products 


14.  What  is  the  vendor's  track  record  with  implementing  the  COTS  product(s)  within 
their  cost  proposal?  (Impossible  to  answer) 

a.  Below  total  life  cycle  cost  estimate 

b.  Met  total  life  cycle  cost  estimate 

c.  Exceeded  total  life  cycle  cost  estimate 

15.  How  financially  stable  is  the  vendor? 

a.  Solid  financial  situation 

b.  Mixed  financial  picture,  may  have  strong  revenue  but  no  profit  margin 

c.  Financial  problems,  such  as  poor  credit,  low  revenues  and  low  profit 
margin 

16.  To  what  extent  does  your  acquisition  approach  include  an  understanding  of  the 
vendor's  future  plans  for  the  COTS  product(s)? 

a.  Statement  of  direction  for  the  product,  including  planned 
enhancements  and  release  dates,  has  been  received 

b.  Discussions  have  been  conducted  with  vendor  regarding  future  direction, 
but  no  plans  have  been  received  in  writing 

c.  No  discussion  with  vendor  regarding  future  direction 


17.  Have  data  rights  been  properly  negotiated  with  the  vendors? 

a.  All  data  rights  negotiated  into  the  contract 

b.  Many  data  rights  have  been  negotiated  into  the  contract 

c.  No  data  rights  have  been  negotiated  into  the  contract 
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18.  Have  cost-effective  licensing  agreements  been  worked  out  with  the  vendors? 

a.  All  licensing  agreements  negotiated  into  cost 

b.  Many  licensing  agreements  negotiated  into  cost 

c.  Uncertain  what  licensing  agreements  are  needed 


Technology 

1 .  Is  the  COTS  application  package(s)  a  totally  new  system  for  the  organization? 

a.  System  is  a  replacement 

b.  Components  of  the  system  are  new 

c.  New  system 

2.  To  adequately  address  your  organization’s  needs,  what  is  the  level  of 
customization  required  for  the  COTS  product(s)  baseline? 

a.  No  customization  necessary 

b.  Some  customization  necessary 

c.  Much  customization  necessary 

3.  How  do  the  COTS  application  package(s)  "fit"  with  the  organization's  existing 
and  planned  architecture? 

a.  Good  lit 

b.  May  fit 

c.  Not  a  fit 
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4. 


Is  the  COTS  product(s)  view  as  a  time-tested,  mature  product? 


a.  Very  mature 

b.  Somewhat  mature 

c.  New  or  immature 


5.  How  many  functions  (e.g.,  accounting,  procurement)  are  supported  by  the  COTS 
application  package(s)? 

a.  Single  function 

b.  Few  functions 

c.  Many  functions 


6.  How  would  you  describe  the  complexity  of  the  interfaces  between  the  COTS 
product(s)  and  other  systems? 

a.  Simple,  easy  to  understand 

b.  Somewhat  complex 

c.  Very  complex,  difficult 


7.  How  many  systems  interfaces  must  remain  unchanged  after  the  implementation  of 
the  COTS  product(s)? 

a.  Few 

b.  Some 

c.  Many 


8.  How  would  you  describe  the  sufficiency  of  documentation  supporting  the 
system(s)  with  which  the  COTS  application  package(s)  will  interface? 

a.  Extremely  high  quality,  thorough  documentation 

b.  Adequate,  some  documentation 

c.  Poor  documentation  or  does  not  exist 
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9.  To  what  extent  has  your  organization  tested  COTS  application  package(s)  in  your 
environment? 

a.  Conducted  extensive  testing 

b.  Conducted  some  testing 

c.  Have  not  conducted  any  testing 


10.  Do  the  security  features  included  in  the  COTS  product(s)  need  modification  to 
meet  your  organization's  requirements? 

a.  Meets  security  requirements,  no  modification  needed 

b.  Meets  most  security  requirements,  some  modification  needed 

c.  Will  not  handle  security  requirements,  extensive  modification  needed 


1 1 .  How  flexible  is  the  design  of  the  COTS  product(s)  to  allow  for  future  changes  in 
functionality? 

a.  Very  flexible — product  functions  can  be  easily  separated  to  be 
modified 

b.  Moderately  flexible — product  functions  can  be  separated  to  be  modified 

c.  Not  flexible — product  functions  can  not  be  separated  to  be  modified 


12.  What  is  the  reliability  history  of  the  COTS  product? 

a.  Product  is  stable  and  has  proven  itself  over  time  with  its  customer 
base 

b.  Product  has  occasional  errors  but  none  will  result  in  data  loss  or  other 
critical  problem 

c.  Product  has  errors  that  result  in  data  loss,  work  lost,  system  crashes,  etc. 


Responses  in  Technology  Section: 

#  a  Vl_  x  1  =J2 
#b  _0_  x  2  =  _0 

#  c  _0_  x  3  =  _0 

Total  =  12 
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Implementation/Logistics  Support 


1.  Has  your  organization  examined  and  applied  the  lessons  learned  from  other 
organizations  that  implemented  the  COTS  application  package(s)? 

a.  Yes — relevant  lessons  learned  have  been  incorporated  into  the 
implementation  plan 

b.  Somewhat — past  projects  have  been  discussed  by  the  project  team 

c.  No — have  not  gathered  any  infonnation  regarding  other  implementations 

2.  How  will  your  organization  measure  the  impact  and  effectiveness  of  the  COTS 
product(s)? 

a.  Comprehensive  performance  measures  (including  cost,  time  spent  on 
each  activity,  etc.)  have  been  established 

b.  Performance  measures  have  been  discussed  but  not  finalized 

c.  No  discussion  of  performance  measures 


3.  What  sort  of  testing  approach  is  planned  for  the  COTS  product(s)? 

a.  Designed  specifically  for  a  COTS  implementation 

b.  Combines  traditional  systems  development  testing  with  COTS-specific 
testing 

c.  Designed  for  traditional  systems  development  activities 


4.  How  would  you  describe  your  organization's  ability  to  support  new  releases  of  the 
COTS  product(s)? 

a.  Sufficient — staffing  plan  for  ongoing  support  of  the  COTS  application 
package(s)  has  been  developed 

b.  Moderate — staffing  needs  have  been  identified,  but  plan  has  not  been 
finalized 

c.  Minimal — no  staff  resources  are  available  after  the  initial  implementation 
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5.  How  has  the  organization  prepared  for  the  possibility  that  the  COTS  application 
package(s)  vendor  goes  out  of  business  or  discontinues  support  for  the  product? 

a.  Contingency  plan  finalized  and  ready  to  implement 

b.  Possibility  discussed,  but  have  no  finalized  plan 

c.  Possibility  not  discussed,  no  contingency  plan  being  developed 

6.  How  would  you  describe  the  run  time  perfonnance  of  the  COTS  product(s)  in 
your  environment? 

a.  Very  efficient 

b.  Moderately  efficient 

c.  Not  efficient 

7.  Does  the  run  time  perfonnance  of  the  COTS  application  package(s)  meet  the 
organization's  performance  needs? 

a.  Efficiently  supports  the  number  and  location  of  users 

b.  Supports  needs  with  performance  degradation 

c.  Does  not  support  needs 

8.  How  do  other  users  of  the  COTS  product  describe  their  satisfaction  with 
availability  of  the  vendor  staff? 

a.  Very  satisfied,  easy  to  access  key  personnel  at  vendor 

b.  Somewhat  satisfied,  can  access  key  personnel  some  of  the  time 

c.  Unsatisfied,  access  to  key  personnel  is  difficult 

9.  What  training  is  needed  to  operate  and  maintain  the  COTS  product? 

a.  No  training 

b.  Some  training 

c.  Extensive  training 
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10.  What  training  sources  are  available  to  the  customers? 

a.  Extensive  training  resources 

b.  Some  training  resources 

c.  No  training  resources 

1 1 .  How  much  experience  does  other  support  contractors  serving  your  organization  in 
functions  affected  by  the  COTS  implementation  have  with  the  COTS  application 
package(s)? 

a.  Extensive  experience 

b.  Some  experience 

c.  No  experience 

Responses  in  Implementation/Logistical  Support  Section: 

#  a  _1_0_  x  1  =  JO 

#  b  J_  x  2  =  _2 

#  c  _0_  x  3  =  _0 

Total  =  12 
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APPENDIX  G  MARINE  CORPS  COMBAT  VEHICLE  TRAINING 
SIMULATOR  QUESTIONNAIRE  RESPONSES 


SECTION  I 

Service:  United  States  Marine  Corps 

Agency  or  Organization: _ Marine  Corps  Systems  Command 

Project/System  Name:  Global  Transportation  and  Engineer  Systems 

Which  category  best  describes  your  main  job  function  in  your  organization? 

a.  Management 

b.  System  analysis  or  design 

c.  Application  or  system  programming 

Have  you  participated  in  selecting  COTS  software  components  that  where  later 
adapted  or  integrated  into  your  project/system?  How  many  times?  No. 

How  long  is  your  work  experience  with  building  system  from  COTS 
components?  _5 Years 

Please  state  any  good  practices,  or  lessons  you  have  learnt  from  past  experience 
when  acquiring  and  developing  systems  using  COTS  software. 

•  Combined  Synopsis/Solicitation  are  great  tools 

SECTION  II 

Questions  are  organized  around  the  three  broad  areas  of  implementing  a  COTS 
solution  as  presented  above.  Each  question  prompts  you,  the  respondent,  to  think  about 
key  factors  for  a  successful  COTS  application  package  implementation.  You  should 
carefully  consider  your  answer  in  terms  of  how  it  pertains  to  projects  within  your  own 
organization. 

Answers  to  each  question  are  provided  by  the  choice  a,  b  or  c,  which  correlate  to 
the  three  levels  of  risk:  low,  medium  and  high,  respectively.  Please  record  your  answers 
to  the  questionnaire  directly  on  this  form.  We  ask  that  you  make  the  best  effort  possible 
to  provide  an  answer  to  all  the  questions.  If  you  are  unsure  of  an  answer,  or  feel  a 
question  does  not  apply  to  your  project,  please  indicate  so  rather  than  leaving  a  question 
blank. 
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Process 


1 .  How  well  are  the  requirements  for  your  system/program  documented? 

a.  Thoroughly — comprehensive,  current  documentation  exists 

b.  Moderately  well — comprehensive  documentation  exists,  but  has  not 
been  updated  recently 

c.  Poorly — minimal  documentation  exists 

2.  Because  specific  requirements  are  associated  with  each  COTS  application 
package(s),  how  would  you  describe  the  relationship  between  the  specifications  of  the 
COTS  product(s)  and  the  requirements  of  your  system/program? 

a.  Ideal — great  fit,  fully  meets  requirements 

b.  Satisfactory — acceptable  lit,  meets  most  requirements 

c.  Unsatisfactory — marginal  lit,  must  be  modified  to  meet  requirements 

3.  How  many  COTS  product(s)  can  accommodate  your  system/program 
requirements? 

a.  Many 

b.  Some 

c.  Few 


4.  How  would  you  describe  the  process  by  which  your  organization  will  implement 
new  requirements  after  the  initial  implementation  of  the  COTS  product(s)? 

a.  Well-defined,  proven  process  has  been  established  to  evaluate  and 
implement  new  requirements  (e.g.,  configuration  control  board) 

b.  Process  for  evaluating  and  implementing  new  requirements  has  been 
discussed,  but  not  solidified 

c.  No  process  exists  for  evaluating  and  implementing  new  requirements 
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5.  How  would  you  describe  your  system/program’s  ability  to  adapt  to  the  new 
requirements  supported  by  the  COTS  product(s)? 

a.  Very  able — there  is  a  general  understanding  that  the  new 
requirements  would  enhance  organization's  operation 

b.  Somewhat  able — there  is  a  general  understanding  that  the  new 
requirements  would  not  enhance  or  deter  organization's  operation 

c.  Not  able — there  is  a  general  understanding  that  the  new  requirements 
would  deter  organization's  operation 


6.  How  was  the  COTS  product  evaluated  and  selected? 

a.  Well-defined,  proven  process  has  been  established  to  evaluate  and  select 
cots  product 

b.  Process  for  evaluating  and  selecting  COTS  products  have  been 
discussed,  but  not  solidified 

c.  Ad  hoc,  no  process  exists  for  evaluating  and  implementing  new 
requirements 


7.  What  is  the  vendor's  experience  with  implementing  the  COTS  product(s)  in 
organizations  of  a  size  similar  to  yours? 

a.  Extensive  experience,  established  company  with  quality  workforce 
and  facilities 

b.  Some  experience 

c.  No  experience,  company  is  start-up  and  situation  is  highly  dynamic 


8.  How  has  the  vendor  performed  in  the  integration  of  the  COTS  application 
package(s)  elsewhere? 

a.  Excellent  past  perfonnance 

b.  Good  past  performance 

c.  Poor  or  unknown  past  performance 
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9.  What  is  the  vendor's  experience  with  implementing  the  considered  COTS 
product(s)  in  organizations  of  a  management  structure  similar  to  yours? 

a.  Extensive  experience,  established  company  with  quality  workforce  and 
facilities 

b.  Some  experience  -  first  major  DOD  implementation 

c.  No  experience,  company  is  start-up  and  situation  is  highly  dynamic 


10.  How  would  you  describe  the  operational  control  of  the  organization  affected  by 
the  COTS  product(s)  implementation? 

a.  Centralized 

b.  Combination  of  centralized  and  decentralized 

c.  Decentralized 


1 1 .  How  would  you  describe  the  sufficiency  of  skilled  staff  in  the  system/program 
affected  by  the  COTS  application  package(s)  implementation? 

a.  Sufficiently  staffed  and  skilled 

b.  Minimally  staffed  and  skilled 

c.  Insufficiently  staffed  and  skilled 


12.  How  much  experience  does  the  COTS  implementation  project  team  have  with  the 
COTS  product(s)? 

a.  Extensive  experience 

b.  Some  experience 

c.  No  experience 
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13.  How  much  experience  does  the  project  team  have  with  implementation  of  other 
COTS  products? 

a.  Experienced  with  many  COTS  products 

b.  Experienced  with  a  few  COTS  products 

c.  Experienced  with  no  other  COTS  products 


14.  What  is  the  vendor's  track  record  with  implementing  the  COTS  product(s)  within 
their  cost  proposal? 

a.  Below  total  life  cycle  cost  estimate 

b.  Met  total  life  cycle  cost  estimate 

c.  Exceeded  total  life  cycle  cost  estimate 

15.  How  financially  stable  is  the  vendor? 

a.  Solid  financial  situation 

b.  Mixed  financial  picture,  may  have  strong  revenue  but  no  profit 
margin 

c.  Financial  problems,  such  as  poor  credit,  low  revenues  and  low  profit 
margin 

16.  To  what  extent  does  your  acquisition  approach  include  an  understanding  of  the 
vendor's  future  plans  for  the  COTS  product(s)? 

a.  Statement  of  direction  for  the  product,  including  planned  enhancements 
and  release  dates,  has  been  received 

b.  Discussions  have  been  conducted  with  vendor  regarding  future 
direction,  but  no  plans  have  been  received  in  writing 

c.  No  discussion  with  vendor  regarding  future  direction 


17.  Have  data  rights  been  properly  negotiated  with  the  vendors? 

a.  All  data  rights  negotiated  into  the  contract 

b.  Many  data  rights  have  been  negotiated  into  the  contract 

No  data  rights  have  been  negotiated  into  the  contract 
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c. 


18.  Have  cost-effective  licensing  agreements  been  worked  out  with  the  vendors? 

a.  All  licensing  agreements  negotiated  into  cost 

b.  Many  licensing  agreements  negotiated  into  cost 

c.  Uncertain  what  licensing  agreements  are  needed 


Technology 

1 .  Is  the  COTS  application  package(s)  a  totally  new  system  for  the  organization? 

a.  System  is  a  replacement 

b.  Components  of  the  system  are  new 

c.  New  system 

2.  To  adequately  address  your  organization’s  needs,  what  is  the  level  of 
customization  required  for  the  COTS  product(s)  baseline? 

a.  No  customization  necessary 

b.  Some  customization  necessary 

c.  Much  customization  necessary 

3.  How  do  the  COTS  application  package(s)  "fit"  with  the  organization's  existing 
and  planned  architecture? 

a.  Good  fit 

b.  May  fit 

c.  Not  a  fit 
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4. 


Is  the  COTS  product(s)  view  as  a  time-tested,  mature  product? 


a.  Very  mature 

b.  Somewhat  mature 

c.  New  or  immature 


5.  How  many  functions  (e.g.,  accounting,  procurement)  are  supported  by  the  COTS 
application  package(s)? 

a.  Single  function 

b.  Few  functions 

c.  Many  functions 


6.  How  would  you  describe  the  complexity  of  the  interfaces  between  the  COTS 
product(s)  and  other  systems? 

a.  Simple,  easy  to  understand 

b.  Somewhat  complex 

c.  Very  complex,  difficult 


7.  How  many  systems  interfaces  must  remain  unchanged  after  the  implementation  of 
the  COTS  product(s)? 

a.  Few 

b.  Some 

c.  Many 


8.  How  would  you  describe  the  sufficiency  of  documentation  supporting  the 
system(s)  with  which  the  COTS  application  package(s)  will  interface? 

a.  Extremely  high  quality,  thorough  documentation 

b.  Adequate,  some  documentation 

c.  Poor  documentation  or  does  not  exist 
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9.  To  what  extent  has  your  organization  tested  COTS  application  package(s)  in  your 
environment? 

a.  Conducted  extensive  testing 

b.  Conducted  some  testing 

c.  Have  not  conducted  any  testing 


10.  Do  the  security  features  included  in  the  COTS  product(s)  need  modification  to 
meet  your  organization's  requirements? 

a.  Meets  security  requirements,  no  modification  needed 

b.  Meets  most  security  requirements,  some  modification  needed 

c.  Will  not  handle  security  requirements,  extensive  modification  needed 


1 1 .  How  flexible  is  the  design  of  the  COTS  product(s)  to  allow  for  future  changes  in 
functionality? 

a.  Very  flexible — product  functions  can  be  easily  separated  to  be 
modified 

b.  Moderately  flexible — product  functions  can  be  separated  to  be  modified 

c.  Not  flexible — product  functions  can  not  be  separated  to  be  modified 


12.  What  is  the  reliability  history  of  the  COTS  product? 

a.  Product  is  stable  and  has  proven  itself  over  time  with  its  customer 
base 

b.  Product  has  occasional  errors  but  none  will  result  in  data  loss  or  other 
critical  problem 

c.  Product  has  errors  that  result  in  data  loss,  work  lost,  system  crashes,  etc. 


Responses  in  Technology  Section: 

#a^_x1  =  _5 

#  b  _5_  x  2  =  _10 

#  c  _2_  x  3  =  _6 

Total  =  21 
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Implementation/Logistics  Support 


1.  Has  your  organization  examined  and  applied  the  lessons  learned  from  other 
organizations  that  implemented  the  COTS  application  package(s)? 

a.  Yes — relevant  lessons  learned  have  been  incorporated  into  the 
implementation  plan 

b.  Somewhat — past  projects  have  been  discussed  by  the  project  team 

c.  No — have  not  gathered  any  information  regarding  other 
implementations 

2.  How  will  your  organization  measure  the  impact  and  effectiveness  of  the  COTS 
product(s)? 

a.  Comprehensive  performance  measures  (including  cost,  time  spent  on  each 
activity,  etc.)  have  been  established 

b.  Performance  measures  have  been  discussed  but  not  finalized 

c.  No  discussion  of  performance  measures 

3.  What  sort  of  testing  approach  is  planned  for  the  COTS  product(s)? 

a.  Designed  specifically  for  a  COTS  implementation 

b.  Combines  traditional  systems  development  testing  with  COTS-specific 
testing 

c.  Designed  for  traditional  systems  development  activities 


4.  How  would  you  describe  your  organization's  ability  to  support  new  releases  of  the 
COTS  product(s)? 

a.  Sufficient — staffing  plan  for  ongoing  support  of  the  COTS  application 
package(s)  has  been  developed 

b.  Moderate — staffing  needs  have  been  identified,  but  plan  has  not  been 
finalized 

c.  Minimal — no  staff  resources  are  available  after  the  initial  implementation 
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5.  How  has  the  organization  prepared  for  the  possibility  that  the  COTS  application 
package(s)  vendor  goes  out  of  business  or  discontinues  support  for  the  product? 

a.  Contingency  plan  finalized  and  ready  to  implement 

b.  Possibility  discussed,  but  have  no  finalized  plan 

c.  Possibility  not  discussed,  no  contingency  plan  being  developed 


6.  How  would  you  describe  the  run  time  perfonnance  of  the  COTS  product(s)  in 
your  environment? 

a.  Very  efficient 

b.  Moderately  efficient 

c.  Not  efficient 


7.  Does  the  run  time  perfonnance  of  the  COTS  application  package(s)  meet  the 
organization's  performance  needs? 

a.  Efficiently  supports  the  number  and  location  of  users 

b.  Supports  needs  with  performance  degradation 

c.  Does  not  support  needs 


8.  How  do  other  users  of  the  COTS  product  describe  their  satisfaction  with 
availability  of  the  vendor  staff? 

a.  Very  satisfied,  easy  to  access  key  personnel  at  vendor 

b.  Somewhat  satisfied,  can  access  key  personnel  some  of  the  time 

c.  Unsatisfied,  access  to  key  personnel  is  difficult 


9.  What  training  is  needed  to  operate  and  maintain  the  COTS  product? 


a. 

b. 


c. 


No  training 
Some  training 

Extensive  training 
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10.  What  training  sources  are  available  to  the  customers? 

a.  Extensive  training  resources 

b.  Some  training  resources 

c.  No  training  resources  (very  little) 

1 1 .  How  much  experience  does  other  support  contractors  serving  your  organization  in 
functions  affected  by  the  COTS  implementation  have  with  the  COTS  application 
package(s)? 

a.  Extensive  experience 

b.  Some  experience 

c.  No  experience 

Responses  in  Implementation/Logistical  Support  Section: 

#  a  1x1  =_2 

#  b  _4_x2  =  _8 

#  c  _5_  x  3  =  _15 

Total  =  25 
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APPENDIX  H  ARMY  COMMON  SOFTWARE  PROGRAM 
QUESTIONNAIRE  RESPONSES 


SECTION  I 

Service:  U.S.  Army 

Agency  or  Organization: _ PM  Ground  Combat  Command  &  Control  (GC  C2) 

Project/System  Name:  Common  Software 

Which  category  best  describes  your  main  job  function  in  your  organization? 

a.  Management 

b.  System  analysis  or  design 

c.  Application  or  system  programming 

Have  you  participated  in  selecting  COTS  software  components  that  where  later 
adapted  or  integrated  into  your  project/system?  How  many  times?  Yes,  one  time. 

How  long  is  your  work  experience  with  building  system  from  COTS 
components?  _2 Years 

Please  state  any  good  practices,  or  lessons  you  have  learnt  from  past  experience 
when  acquiring  and  developing  systems  using  COTS  software. 

•  Never  rely  on  a  single  vendor  for  critical  functionality,  always  have 
alternate  products  lined  up 

•  Carefully  consider  the  likelihood  that  the  vendor  will  not  be  there  to 
support  it  in  the  future 

SECTION  II 

Questions  are  organized  around  the  three  broad  areas  of  implementing  a  COTS 
solution  as  presented  above.  Each  question  prompts  you,  the  respondent,  to  think  about 
key  factors  for  a  successful  COTS  application  package  implementation.  You  should 
carefully  consider  your  answer  in  terms  of  how  it  pertains  to  projects  within  your  own 
organization. 

Answers  to  each  question  are  provided  by  the  choice  a,  b  or  c,  which  correlate  to 
the  three  levels  of  risk:  low,  medium  and  high,  respectively.  Please  record  your  answers 
to  the  questionnaire  directly  on  this  form.  We  ask  that  you  make  the  best  effort  possible 
to  provide  an  answer  to  all  the  questions.  If  you  are  unsure  of  an  answer,  or  feel  a 
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question  does  not  apply  to  your  project,  please  indicate  so  rather  than  leaving  a  question 
blank. 

Process 

1 .  How  well  are  the  requirements  for  your  system/program  documented? 

a.  Thoroughly — comprehensive,  current  documentation  exists 

b.  Moderately  well — comprehensive  documentation  exists,  but  has  not 
been  updated  recently 

c.  Poorly — minimal  documentation  exists 

2.  Because  specific  requirements  are  associated  with  each  COTS  application 
package(s),  how  would  you  describe  the  relationship  between  the  specifications  of  the 
COTS  product(s)  and  the  requirements  of  your  system/program? 

a.  Ideal — great  fit,  fully  meets  requirements 

b.  Satisfactory — acceptable  lit,  meets  most  requirements 

c.  Unsatisfactory — marginal  fit,  must  be  modified  to  meet  requirements 

3.  How  many  COTS  product(s)  can  accommodate  your  system/program 
requirements? 

a.  Many 

b.  Some 

c.  Few 


4.  How  would  you  describe  the  process  by  which  your  organization  will  implement 
new  requirements  after  the  initial  implementation  of  the  COTS  product(s)? 

a.  Well-defined,  proven  process  has  been  established  to  evaluate  and 
implement  new  requirements  (e.g.,  configuration  control  board) 

b.  Process  for  evaluating  and  implementing  new  requirements  has  been 
discussed,  but  not  solidified 

c.  No  process  exists  for  evaluating  and  implementing  new  requirements 
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5.  How  would  you  describe  your  system/program’s  ability  to  adapt  to  the  new 
requirements  supported  by  the  COTS  product(s)? 

a.  Very  able — there  is  a  general  understanding  that  the  new  requirements 
would  enhance  organization's  operation 

b.  Somewhat  able — there  is  a  general  understanding  that  the  new 
requirements  would  not  enhance  or  deter  organization's  operation 

c.  Not  able — there  is  a  general  understanding  that  the  new  requirements 
would  deter  organization's  operation 


6.  How  was  the  COTS  product  evaluated  and  selected? 

a.  Well-defined,  proven  process  has  been  established  to  evaluate  and  select 
cots  product 

b.  Process  for  evaluating  and  selecting  COTS  products  have  been 
discussed,  but  not  solidified 

c.  Ad  hoc,  no  process  exists  for  evaluating  and  implementing  new 
requirements 


7.  What  is  the  vendor's  experience  with  implementing  the  COTS  product(s)  in 
organizations  of  a  size  similar  to  yours? 

a.  Extensive  experience,  established  company  with  quality  workforce 
and  facilities 

b.  Some  experience 

c.  No  experience,  company  is  start-up  and  situation  is  highly  dynamic 


8.  How  has  the  vendor  performed  in  the  integration  of  the  COTS  application 
package(s)  elsewhere? 

a.  Excellent  past  performance 

b.  Good  past  performance 

c.  Poor  or  unknown  past  performance 
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9.  What  is  the  vendor's  experience  with  implementing  the  considered  COTS 
product(s)  in  organizations  of  a  management  structure  similar  to  yours? 

a.  Extensive  experience,  established  company  with  quality  workforce 
and  facilities 

b.  Some  experience  -  first  major  DOD  implementation 

c.  No  experience,  company  is  start-up  and  situation  is  highly  dynamic 


10.  How  would  you  describe  the  operational  control  of  the  organization  affected  by 
the  COTS  product(s)  implementation? 

a.  Centralized 

b.  Combination  of  centralized  and  decentralized 

c.  Decentralized 


1 1 .  How  would  you  describe  the  sufficiency  of  skilled  staff  in  the  system/program 
affected  by  the  COTS  application  package(s)  implementation? 

a.  Sufficiently  staffed  and  skilled 

b.  Minimally  staffed  and  skilled 

c.  Insufficiently  staffed  and  skilled 

12.  How  much  experience  does  the  COTS  implementation  project  team  have  with  the 
COTS  product(s)? 

a.  Extensive  experience 

b.  Some  experience 

c.  No  experience 
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13.  How  much  experience  does  the  project  team  have  with  implementation  of  other 
COTS  products? 

a.  Experienced  with  many  COTS  products 

b.  Experienced  with  a  few  COTS  products 

c.  Experienced  with  no  other  COTS  products 


14.  What  is  the  vendor's  track  record  with  implementing  the  COTS  product(s)  within 
their  cost  proposal?  (Impossible  to  answer) 

a.  Below  total  life  cycle  cost  estimate 

b.  Met  total  life  cycle  cost  estimate 

c.  Exceeded  total  life  cycle  cost  estimate 

15.  How  financially  stable  is  the  vendor? 

a.  Solid  financial  situation 

b.  Mixed  financial  picture,  may  have  strong  revenue  but  no  profit  margin 

c.  Financial  problems,  such  as  poor  credit,  low  revenues  and  low  profit 
margin 

16.  To  what  extent  does  your  acquisition  approach  include  an  understanding  of  the 
vendor's  future  plans  for  the  COTS  product(s)? 

a.  Statement  of  direction  for  the  product,  including  planned 
enhancements  and  release  dates,  has  been  received 

b.  Discussions  have  been  conducted  with  vendor  regarding  future  direction, 
but  no  plans  have  been  received  in  writing 

c.  No  discussion  with  vendor  regarding  future  direction 


17.  Have  data  rights  been  properly  negotiated  with  the  vendors? 

a.  All  data  rights  negotiated  into  the  contract 

b.  Many  data  rights  have  been  negotiated  into  the  contract 

c.  No  data  rights  have  been  negotiated  into  the  contract 
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18.  Have  cost-effective  licensing  agreements  been  worked  out  with  the  vendors? 

a.  All  licensing  agreements  negotiated  into  cost 

b.  Many  licensing  agreements  negotiated  into  cost 

c.  Uncertain  what  licensing  agreements  are  needed 


Technology 

1 .  Is  the  COTS  application  package(s)  a  totally  new  system  for  the  organization? 

a.  System  is  a  replacement 

b.  Components  of  the  system  are  new 

c.  New  system 

2.  To  adequately  address  your  organization’s  needs,  what  is  the  level  of 
customization  required  for  the  COTS  product(s)  baseline? 

a.  No  customization  necessary 

b.  Some  customization  necessary 

c.  Much  customization  necessary 

3.  How  do  the  COTS  application  package(s)  "fit"  with  the  organization's  existing 
and  planned  architecture? 

a.  Good  lit 

b.  May  fit 

c.  Not  a  fit 
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4.  Is  the  COTS  product(s)  view  as  a  time-tested,  mature  product? 

a.  Very  mature 

b.  Somewhat  mature 

c.  New  or  immature 

5.  How  many  functions  (e.g.,  accounting,  procurement)  are  supported  by  the  COTS 
application  package(s)? 

a.  Single  function 

b.  Few  functions 

c.  Many  functions 


6.  How  would  you  describe  the  complexity  of  the  interfaces  between  the  COTS 
product(s)  and  other  systems? 

a.  Simple,  easy  to  understand 

b.  Somewhat  complex 

c.  Very  complex,  difficult 


7.  How  many  systems  interfaces  must  remain  unchanged  after  the  implementation  of 
the  COTS  product(s)? 

a.  Few 

b.  Some 

c.  Many 


8.  How  would  you  describe  the  sufficiency  of  documentation  supporting  the 
system(s)  with  which  the  COTS  application  package(s)  will  interface? 

a.  Extremely  high  quality,  thorough  documentation 

b.  Adequate,  some  documentation 

c.  Poor  documentation  or  does  not  exist 
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9.  To  what  extent  has  your  organization  tested  COTS  application  package(s)  in  your 
environment? 

a.  Conducted  extensive  testing 

b.  Conducted  some  testing 

c.  Have  not  conducted  any  testing 

10.  Do  the  security  features  included  in  the  COTS  product(s)  need  modification  to 
meet  your  organization's  requirements? 

a.  Meets  security  requirements,  no  modification  needed 

b.  Meets  most  security  requirements,  some  modification  needed 

c.  Will  not  handle  security  requirements,  extensive  modification  needed 

1 1 .  How  flexible  is  the  design  of  the  COTS  product(s)  to  allow  for  future  changes  in 
functionality? 

a.  Very  flexible — product  functions  can  be  easily  separated  to  be 
modified 

b.  Moderately  flexible — product  functions  can  be  separated  to  be  modified 

c.  Not  flexible — product  functions  can  not  be  separated  to  be  modified 


12.  What  is  the  reliability  history  of  the  COTS  product? 

a.  Product  is  stable  and  has  proven  itself  over  time  with  its  customer  base 

b.  Product  has  occasional  errors  but  none  will  result  in  data  loss  or  other 
critical  problem 

c.  Product  has  errors  that  result  in  data  loss,  work  lost,  system  crashes,  etc. 


Responses  in  Technology  Section: 

#  a  4_x  1  =  _4 

#  b  _6_x2  =  J2 

#  c  _2_  x  3  =  _6 

Total  =  22 
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Implementation/Logistics  Support 


1.  Has  your  organization  examined  and  applied  the  lessons  learned  from  other 
organizations  that  implemented  the  COTS  application  package(s)? 

a.  Yes — relevant  lessons  learned  have  been  incorporated  into  the 
implementation  plan 

b.  Somewhat — past  projects  have  been  discussed  by  the  project  team 

c.  No — have  not  gathered  any  information  regarding  other 
implementations 

2.  How  will  your  organization  measure  the  impact  and  effectiveness  of  the  COTS 
product(s)? 

a.  Comprehensive  performance  measures  (including  cost,  time  spent  on  each 
activity,  etc.)  have  been  established 

b.  Performance  measures  have  been  discussed  but  not  finalized 

c.  No  discussion  of  performance  measures 


3.  What  sort  of  testing  approach  is  planned  for  the  COTS  product(s)? 

a.  Designed  specifically  for  a  COTS  implementation 

b.  Combines  traditional  systems  development  testing  with  COTS-specific 
testing 

c.  Designed  for  traditional  systems  development  activities 


4.  How  would  you  describe  your  organization's  ability  to  support  new  releases  of  the 
COTS  product(s)? 

a.  Sufficient — staffing  plan  for  ongoing  support  of  the  COTS  application 
package(s)  has  been  developed 

b.  Moderate — staffing  needs  have  been  identified,  but  plan  has  not  been 
finalized 

c.  Minimal — no  staff  resources  are  available  after  the  initial  implementation 
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5.  How  has  the  organization  prepared  for  the  possibility  that  the  COTS  application 
package(s)  vendor  goes  out  of  business  or  discontinues  support  for  the  product? 

a.  Contingency  plan  finalized  and  ready  to  implement 

b.  Possibility  discussed,  but  have  no  finalized  plan 

c.  Possibility  not  discussed,  no  contingency  plan  being  developed 

6.  How  would  you  describe  the  run  time  perfonnance  of  the  COTS  product(s)  in 
your  environment? 

a.  Very  efficient 

b.  Moderately  efficient 

c.  Not  efficient 

7.  Does  the  run  time  perfonnance  of  the  COTS  application  package(s)  meet  the 
organization's  performance  needs? 

a.  Efficiently  supports  the  number  and  location  of  users 

b.  Supports  needs  with  performance  degradation 

c.  Does  not  support  needs 

8.  How  do  other  users  of  the  COTS  product  describe  their  satisfaction  with 
availability  of  the  vendor  staff? 

a.  Very  satisfied,  easy  to  access  key  personnel  at  vendor 

b.  Somewhat  satisfied,  can  access  key  personnel  some  of  the  time 

c.  Unsatisfied,  access  to  key  personnel  is  difficult 

9.  What  training  is  needed  to  operate  and  maintain  the  COTS  product? 

a.  No  training 

b.  Some  training 

c.  Extensive  training 
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10.  What  training  sources  are  available  to  the  customers? 

a.  Extensive  training  resources 

b.  Some  training  resources 

c.  No  training  resources 

1 1 .  How  much  experience  does  other  support  contractors  serving  your  organization  in 
functions  affected  by  the  COTS  implementation  have  with  the  COTS  application 
package(s)? 

a.  Extensive  experience 

b.  Some  experience 

c.  No  experience 

Responses  in  Implementation/Logistical  Support  Section: 

#  a  _6_  x  1  =  _6 

#  b  J_  x  2  =  _2 

#  c  _4_  x  3  =  _12 

Total  =  20 
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